One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

• URL: http://arxiv.org/abs/2603.03291v1
• Date: Fri, 06 Feb 2026 00:11:37 GMT
• Title: One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
• Abstract summary: Reward models (RMs) are crucial for online alignment of language models with human preferences.<n>We measure biases in five high-quality RMs, including the state-of-the-art.<n>We propose a simple post-hoc intervention to mitigate low-complexity biases that arise from spurious correlations.
• Score: 32.15734141265721
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences. However, RM-based preference-tuning is vulnerable to reward hacking, whereby LM policies learn undesirable behaviors from flawed RMs. By systematically measuring biases in five high-quality RMs, including the state-of-the-art, we find that issues persist despite prior work with respect to length, sycophancy, and overconfidence. We also discover new issues related to bias toward model-specific styles and answer-order. We categorize RM failures by complexity and propose a simple post-hoc intervention to mitigate low-complexity biases that arise from spurious correlations. Our proposed mechanistic reward shaping reduces targeted biases without degrading reward quality and while using minimal labeled data. The method is extensible to new biases, model-internal, and generalizes out-of-distribution.

Related papers

• SteerRM: Debiasing Reward Models via Sparse Autoencoders [28.922614335072414]
  SteerRM is the first training-free method for debiasing reward models using Sparse Autoencoder (SAE)-based interventions.<n>Across six reward models on RM-Bench, SteerRM improves Hard-split accuracy by 7.3 points on average while preserving overall performance.
  arXiv  Detail & Related papers  (2026-03-13T08:55:36Z)
• CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [61.75914342638658]
  CDRRM (Contrast-Driven Reward Model) is a framework for high-quality rubric generation and guided preference judgment.<n>This work offers a scalable, interpretable, and data-efficient path for reward modeling.
  arXiv  Detail & Related papers  (2026-03-09T07:15:23Z)
• Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling [49.41422138354821]
  We propose a principled reward modeling framework that integrates non-negative factor analysis into the Bradley-Terry preference model.<n>BNRM represents rewards through a sparse, non-negative latent factor generative process.<n>We show that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.
  arXiv  Detail & Related papers  (2026-02-11T08:14:11Z)
• Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance [46.71732887299883]
  Reward models (RMs) are essential in reinforcement learning from human feedback (RLHF) to align large language models with human values.<n>We introduce a novel information-theoretic debiasing method called textbfDebiasing via textbfInformation optimization for textbfRM (DIR)<n>With theoretical justification from information theory, DIR can handle more sophisticated types of biases with non-linear correlations.
  arXiv  Detail & Related papers  (2025-12-29T13:39:41Z)
• SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models [53.19726629537694]
  Post-training alignment of video generation models with human preferences is a critical goal.<n>Current data collection paradigms, reliant on in-prompt pairwise annotations, suffer from labeling noise.<n>We propose SoliReward, a systematic framework for video RM training.
  arXiv  Detail & Related papers  (2025-12-17T14:28:23Z)
• Reward Model Perspectives: Whose Opinions Do Reward Models Reward? [0.0]
  Reward models (RMs) are central to the alignment of language models (LMs)<n>Our work formalizes a framework for measuring the alignment of opinions captured by RMs.<n>We show that RMs are poorly aligned with several demographic groups and can systematically reward harmful stereotypes.
  arXiv  Detail & Related papers  (2025-10-07T19:13:52Z)
• Bias Fitting to Mitigate Length Bias of Reward Model in RLHF [81.44256822500257]
  Reinforcement Learning from Human Feedback relies on reward models to align large language models with human preferences.<n>We propose FiMi-RM, a framework that autonomously learns and corrects underlying bias patterns.<n> Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution.
  arXiv  Detail & Related papers  (2025-05-19T08:29:28Z)
• Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment [35.80989342492335]
  noisy preferences in human feedback can lead to reward misgeneralization.<n>This paper aims to identify how noisy preferences differ from human-aligned preferences in reward modeling.<n>We propose an online Collaborative Reward Modeling framework to achieve robust preference learning.
  arXiv  Detail & Related papers  (2025-05-15T10:58:20Z)
• Fast Model Debias with Machine Unlearning [54.32026474971696]
  Deep neural networks might behave in a biased manner in many real-world scenarios. Existing debiasing methods suffer from high costs in bias labeling or model re-training. We propose a fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases.
  arXiv  Detail & Related papers  (2023-10-19T08:10:57Z)
• Confronting Reward Model Overoptimization with Constrained RLHF [114.71591361764547]
  We show that correlation between component RMs has a significant effect on the locations of these points. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally expressed by Lagrange multipliers.
  arXiv  Detail & Related papers  (2023-10-06T16:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.