Mitigating Cognitive Bias in RLHF by Altering Rationality

• URL: http://arxiv.org/abs/2605.06895v1
• Date: Thu, 07 May 2026 19:54:25 GMT
• Title: Mitigating Cognitive Bias in RLHF by Altering Rationality
• Abstract summary: We propose an approach to dynamically adjust the rationality parameter beta during reward learning.<n>This approach effectively downweights comparisons that are likely to reflect biased or unreliable judgments.<n> Empirically, we show that this approach learns a more rational downstream model, even when finetuning on datasets with strongly biased preferences.
• Score: 40.54637728189237
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How can we make models robust to even imperfect human feedback? In reinforcement learning from human feedback (RLHF), human preferences over model outputs are used to train a reward model that assigns scalar values to responses. Because these rewards are inferred from pairwise comparisons, this learning depends on an assumed relationship between latent reward differences and observed preferences, typically modeled using a Boltzmann formulation in which a rationality parameter beta informs how consistently preferences reflect reward differences. In practice, beta is typically treated as a fixed constant that reflects assumed uniform annotator reliability. However, human feedback is not this simplistic in practice: real human judgments are shaped by cognitive biases, leading to systematic deviations from reward-consistent behavior that arise contextually. To address this, we treat rationality as context- and annotation-dependent. We design an approach to dynamically adjust the rationality parameter beta during reward learning using an LLM-as-judge to assess the likely presence of cognitive biases. This approach effectively downweights comparisons that are likely to reflect biased or unreliable judgments. Empirically, we show that this approach learns a more rational downstream model, even when finetuning on datasets with strongly biased preferences.

Related papers

• CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks [65.44788139573144]
  We introduce observational reward modeling -- learning reward models with observational user feedback -- as a scalable and cost-effective alternative.<n>CaulRM aims to learn unbiased reward models from observational feedback.<n>Experiments validate that CausalRM effectively learns accurate reward signals from noisy and biased observational feedback.
  arXiv  Detail & Related papers  (2026-03-19T10:37:34Z)
• How RLHF Amplifies Sycophancy [23.213056717401418]
  Large language models often exhibit increased sycophantic behavior after preference-based post-training.<n>We identify an explicit amplification mechanism that causally links optimization against a learned reward to bias in the human preference data used for alignment.<n>We propose a training-time intervention designed to neutralize the amplification mechanism itself.
  arXiv  Detail & Related papers  (2026-02-01T03:46:14Z)
• Debiasing Reward Models by Representation Learning with Guarantees [20.170532007501883]
  We propose a principled framework that mitigates spurious correlation issues in reward models.<n>We show that non-spurious latent variables can be theoretically identified from data, regardless of whether a surrogate for the spurious latent variables is available.<n>Experiments on synthetic and real-world datasets demonstrate that our method effectively mitigates spurious correlation issues and yields more robust reward models.
  arXiv  Detail & Related papers  (2025-10-27T18:37:57Z)
• Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs [51.00909549291524]
  Large language models (LLMs) exhibit cognitive biases.<n>These biases vary across models and can be amplified by instruction tuning.<n>It remains unclear if these differences in biases stem from pretraining, finetuning, or even random noise.
  arXiv  Detail & Related papers  (2025-07-09T18:01:14Z)
• Bias Fitting to Mitigate Length Bias of Reward Model in RLHF [81.44256822500257]
  Reinforcement Learning from Human Feedback relies on reward models to align large language models with human preferences.<n>We propose FiMi-RM, a framework that autonomously learns and corrects underlying bias patterns.<n> Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution.
  arXiv  Detail & Related papers  (2025-05-19T08:29:28Z)
• Reward Modeling with Ordinal Feedback: Wisdom of the Crowd [9.034189257088762]
  Learning a reward model (RM) from human preferences has been an important component in aligning large language models. We propose a framework for learning RMs under ordinal feedback. We prove the statistical benefits of ordinal feedback in terms of reducing the Rademacher complexity.
  arXiv  Detail & Related papers  (2024-11-19T20:17:04Z)
• Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems [82.92678837778358]
  preference-based methods have demonstrated substantial success in empirical applications such as InstructGPT. We show how human bias and uncertainty in feedback modelings can affect the theoretical guarantees of these approaches.
  arXiv  Detail & Related papers  (2023-07-24T17:50:24Z)
• The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types [38.37216644899506]
  We argue that grounding the rationality coefficient in real data for each feedback type has a significant positive effect on reward learning. We find that when learning from a single feedback type, overestimating human rationality can have dire effects on reward accuracy and regret.
  arXiv  Detail & Related papers  (2022-08-23T02:19:10Z)
• Cross Pairwise Ranking for Unbiased Item Recommendation [57.71258289870123]
  We develop a new learning paradigm named Cross Pairwise Ranking (CPR) CPR achieves unbiased recommendation without knowing the exposure mechanism. We prove in theory that this way offsets the influence of user/item propensity on the learning.
  arXiv  Detail & Related papers  (2022-04-26T09:20:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.