source: arxiv artificial intelligence: rater state bias in rlhf preference data: an audit framework
level: research
pairwise preference labels in reinforcement learning from human feedback are meant to capture output quality, but they can also reflect the rater's mental state during annotation. under sustained stressful or distressing conditions, raters' preferences may shift over time. this means preference data can encode rater state alongside judgments about response quality. these shifts are not just ordinary disagreement or random noise. they are state dependent, can be shared across annotators working under similar conditions, and can propagate through reward modeling and policy optimization.
the paper proposes rater state shift as a structured confound in rlhf. it defines key concepts: rater state shift, rater state confound, and correlated rater state bias. the authors develop an audit framework to test for this bias. the framework aims to distinguish state-driven preference changes from genuine quality assessments. by formalizing these ideas, the work provides a way to detect and measure how rater conditions might systematically distort training data.
this bias matters because rlhf is widely used to align large language models with human values. if rater state influences labels, the resulting reward models and policies may inherit unintended preferences. the audit framework offers a tool for practitioners to check their data for such confounds. it encourages more careful data collection practices, such as monitoring rater well-being or adjusting for state effects. ultimately, addressing rater state bias could lead to more reliable and fair ai systems.
why it matters: detecting rater state bias helps ensure ai alignment reflects genuine human preferences rather than annotation conditions, improving model reliability.
source: arxiv artificial intelligence: rater state bias in rlhf preference data: an audit framework