CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks

• URL: http://arxiv.org/abs/2603.18736v1
• Date: Thu, 19 Mar 2026 10:37:34 GMT
• Title: CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks
• Abstract summary: We introduce observational reward modeling -- learning reward models with observational user feedback -- as a scalable and cost-effective alternative.<n>CaulRM aims to learn unbiased reward models from observational feedback.<n>Experiments validate that CausalRM effectively learns accurate reward signals from noisy and biased observational feedback.
• Score: 65.44788139573144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models, current reward modeling heavily relies on experimental feedback data collected from human annotators under controlled and costly conditions. In this work, we introduce observational reward modeling -- learning reward models with observational user feedback (e.g., clicks, copies, and upvotes) -- as a scalable and cost-effective alternative. We identify two fundamental challenges in this setting: (1) observational feedback is noisy due to annotation errors, which deviates it from true user preference; (2) observational feedback is biased by user preference, where users preferentially provide feedback on responses they feel strongly about, which creats a distribution shift between training and inference data. To address these challenges, we propose CausalRM, a causal-theoretic reward modeling framework that aims to learn unbiased reward models from observational feedback. To tackle challenge (1), CausalRM introduces a noise-aware surrogate loss term that is provably equivalent to the primal loss under noise-free conditions by explicitly modeling the annotation error generation process. To tackle challenge (2), CausalRM uses propensity scores -- the probability of a user providing feedback for a given response -- to reweight training samples, yielding a loss function that eliminates user preference bias. Extensive experiments across diverse LLM backbones and benchmark datasets validate that CausalRM effectively learns accurate reward signals from noisy and biased observational feedback and delivers substantial performance improvements on downstream RLHF tasks -- including a 49.2% gain on WildGuardMix and a 32.7% improvement on HarmBench. Code is available on our project website.

Related papers

• User Feedback Provides a Unique Signal that LLMs Can not Detect [25.849928795741672]
  We demonstrate that user feedback is a highly actionable signal for improvement.<n>We show that feedback-informed revisions resolve targeted issues at significantly higher rates than baseline revisions.
  arXiv  Detail & Related papers  (2026-09-02T17:42:44Z)
• ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment [49.93516092785756]
  Current reward modeling is heavily contingent upon experimental feedback data with high collection costs.<n>ImplicitRM aims to learn unbiased reward models from implicit preference data.<n> Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets.
  arXiv  Detail & Related papers  (2026-03-24T13:32:14Z)
• Reward Modeling from Natural Language Human Feedback [77.75758630455357]
  Reinforcement Learning with Verifiable reward (RLVR) on preference data has become the mainstream approach for training Generative Reward Models (GRMs)<n>In this paper, we demonstrate that such binary classification tasks make GRMs susceptible to guessing correct outcomes without sound critiques.<n>We propose Reward Modeling from Natural Language Human Feedback (RM-NLHF), which leverages natural language feedback to obtain process reward signals.
  arXiv  Detail & Related papers  (2026-01-12T09:23:43Z)
• Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking [78.69179041551014]
  We propose an information-theoretic reward modeling framework based on the Information Bottleneck principle.<n>We show that InfoRM filters out preference-irrelevant information to alleviate reward misgeneralization.<n>We also introduce IBL, a distribution-level regularization that penalizes such deviations, effectively expanding the optimization landscape.
  arXiv  Detail & Related papers  (2025-10-15T15:51:59Z)
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning [50.20267980386502]
  We learn a dense, token-level reward model for process supervision directly from expert demonstrations.<n>The learned reasoning reward serves two complementary roles: (i) it provides step-level feedback to optimise a reasoning policy during training; and (ii) it functions at inference as a critic to rerank sampled traces under fixed compute budgets.
  arXiv  Detail & Related papers  (2025-10-02T09:55:26Z)
• Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping [2.427844597259453]
  Reinforcement learning (RL) often struggles with reward misalignment.<n>Human-in-the-loop (HITL) methods can mitigate this issue, but they also introduce biases.<n>We propose two key contributions to address these challenges.
  arXiv  Detail & Related papers  (2025-03-26T03:17:12Z)
• RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation [3.998312409829935]
  We show that Reinforcement Learning from Human Feedback can cause severe, systematic misalignment.<n>We introduce Reinforcement Learning from Hindsight Simulation (RLHS), which presents plausible simulated outcomes to evaluators before eliciting feedback.<n>RLHS substantially improves alignment over RLHF in experiments and human evaluations.
  arXiv  Detail & Related papers  (2025-01-15T06:33:15Z)
• Understanding Impact of Human Feedback via Influence Functions [31.080092511782482]
  In Reinforcement Learning from Human Feedback (RLHF), it is crucial to learn suitable reward models from human feedback.<n>Human feedback can often be noisy, inconsistent, or biased, especially when evaluating complex responses.<n>We propose a compute-efficient approximation method to measure the impact of human feedback on the performance of reward models.
  arXiv  Detail & Related papers  (2025-01-10T08:50:38Z)
• Secrets of RLHF in Large Language Models Part II: Reward Modeling [134.97964938009588]
  We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset. We also introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses.
  arXiv  Detail & Related papers  (2024-01-11T17:56:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.