Debiasing Reward Models via Causally Motivated Inference-Time Intervention

• URL: http://arxiv.org/abs/2604.27495v1
• Date: Thu, 30 Apr 2026 06:49:17 GMT
• Title: Debiasing Reward Models via Causally Motivated Inference-Time Intervention
• Abstract summary: Reward models (RMs) play a central role in aligning large language models with human preferences.<n>We propose causally motivated intervention for mitigating multiple types of biases in RMs at inference time.
• Score: 16.571157391857522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward models (RMs) play a central role in aligning large language models (LLMs) with human preferences. However, RMs are often sensitive to spurious features such as response length. Existing inference-time approaches for mitigating these biases typically focus exclusively on response length, resulting in performance trade-offs. In this paper, we propose causally motivated intervention for mitigating multiple types of biases in RMs at inference time. Our method first identifies neurons whose activations are strongly correlated with predefined bias attributes, and applies neuron-level intervention that suppresses these signals. We evaluate our method on RM benchmarks and observe reductions in sensitivity to spurious features across diverse bias types, without inducing performance trade-offs. Moreover, when used for preference annotation, small RMs (2B and 7B) with our method, which edits less than 2% of all the neurons in RMs, enable LLMs to improve alignment, achieving performance comparable to that of a state-of-the-art 70B RM on AlpacaEval and MT-Bench. Further analysis reveals that bias signals are primarily encoded by neurons in early layers, shedding light on the internal mechanisms of bias exploitation in RMs.

Related papers

• SteerRM: Debiasing Reward Models via Sparse Autoencoders [28.922614335072414]
  SteerRM is the first training-free method for debiasing reward models using Sparse Autoencoder (SAE)-based interventions.<n>Across six reward models on RM-Bench, SteerRM improves Hard-split accuracy by 7.3 points on average while preserving overall performance.
  arXiv  Detail & Related papers  (2026-03-13T08:55:36Z)
• GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models [17.33640761554548]
  We propose GPrune-LLM, a generalization-aware structured pruning framework.<n>We first partition neurons into behavior-consistent modules to localize ranking competition.<n>For modules where activation-based scoring is unreliable, we switch to an activation-independent metric.
  arXiv  Detail & Related papers  (2026-03-12T19:20:37Z)
• One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models [32.15734141265721]
  Reward models (RMs) are crucial for online alignment of language models with human preferences.<n>We measure biases in five high-quality RMs, including the state-of-the-art.<n>We propose a simple post-hoc intervention to mitigate low-complexity biases that arise from spurious correlations.
  arXiv  Detail & Related papers  (2026-02-06T00:11:37Z)
• Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steering [50.63386303357225]
  We propose AdaRAS, a lightweight test-time framework that improves reasoning reliability by selectively intervening on neuron activations.<n>AdaRAS identifies Reasoning-Critical Neurons (RCNs) via a polarity-aware mean-difference criterion and adaptively steers their activations during inference.<n> Experiments on 10 mathematics and coding benchmarks demonstrate consistent improvements, including over 13% gains on AIME-24 and AIME-25.
  arXiv  Detail & Related papers  (2026-01-27T17:53:01Z)
• Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance [46.71732887299883]
  Reward models (RMs) are essential in reinforcement learning from human feedback (RLHF) to align large language models with human values.<n>We introduce a novel information-theoretic debiasing method called textbfDebiasing via textbfInformation optimization for textbfRM (DIR)<n>With theoretical justification from information theory, DIR can handle more sophisticated types of biases with non-linear correlations.
  arXiv  Detail & Related papers  (2025-12-29T13:39:41Z)
• SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models [53.19726629537694]
  Post-training alignment of video generation models with human preferences is a critical goal.<n>Current data collection paradigms, reliant on in-prompt pairwise annotations, suffer from labeling noise.<n>We propose SoliReward, a systematic framework for video RM training.
  arXiv  Detail & Related papers  (2025-12-17T14:28:23Z)
• Detecting and Pruning Prominent but Detrimental Neurons in Large Language Models [68.57424628540907]
  Large language models (LLMs) often develop learned mechanisms specialized to specific datasets.<n>We introduce a fine-tuning approach designed to enhance generalization by identifying and pruning neurons associated with dataset-specific mechanisms.<n>Our method employs Integrated Gradients to quantify each neuron's influence on high-confidence predictions, pinpointing those that disproportionately contribute to dataset-specific performance.
  arXiv  Detail & Related papers  (2025-07-12T08:10:10Z)
• RRM: Robust Reward Model Training Mitigates Reward Hacking [51.12341734942797]
  Reward models (RMs) play a pivotal role in aligning large language models with human preferences.<n>We introduce a causal framework that learns preferences independent of these artifacts.<n>Experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model.
  arXiv  Detail & Related papers  (2024-09-20T01:46:07Z)
• Confronting Reward Model Overoptimization with Constrained RLHF [114.71591361764547]
  We show that correlation between component RMs has a significant effect on the locations of these points. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally expressed by Lagrange multipliers.
  arXiv  Detail & Related papers  (2023-10-06T16:59:17Z)
• Toward Fair Facial Expression Recognition with Improved Distribution Alignment [19.442685015494316]
  We present a novel approach to mitigate bias in facial expression recognition (FER) models. Our method aims to reduce sensitive attribute information such as gender, age, or race, in the embeddings produced by FER models. For the first time, we analyze the notion of attractiveness as an important sensitive attribute in FER models and demonstrate that FER models can indeed exhibit biases towards more attractive faces.
  arXiv  Detail & Related papers  (2023-06-11T14:59:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.