Causally Robust Reward Learning from Reason-Augmented Preference Feedback

• URL: http://arxiv.org/abs/2603.04861v1
• Date: Thu, 05 Mar 2026 06:35:21 GMT
• Title: Causally Robust Reward Learning from Reason-Augmented Preference Feedback
• Abstract summary: We introduce ReCouPLe, a lightweight framework that uses natural language rationales to provide the missing causal signal.<n>Our learned reward model can ground preferences on the articulated reason, aligning better with user intent and generalizing beyond spurious features.
• Score: 4.170114609601759
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Preference-based reward learning is widely used for shaping agent behavior to match a user's preference, yet its sparse binary feedback makes it especially vulnerable to causal confusion. The learned reward often latches onto spurious features that merely co-occur with preferred trajectories during training, collapsing when those correlations disappear or reverse at test time. We introduce ReCouPLe, a lightweight framework that uses natural language rationales to provide the missing causal signal. Each rationale is treated as a guiding projection axis in an embedding space, training the model to score trajectories based on features aligned with that axis while de-emphasizing context that is unrelated to the stated reason. Because the same rationales (e.g., "avoids collisions", "completes the task faster") can appear across multiple tasks, ReCouPLe naturally reuses the same causal direction whenever tasks share semantics, and transfers preference knowledge to novel tasks without extra data or language-model fine-tuning. Our learned reward model can ground preferences on the articulated reason, aligning better with user intent and generalizing beyond spurious features. ReCouPLe outperforms baselines by up to 1.5x in reward accuracy under distribution shifts, and 2x in downstream policy performance in novel tasks. We have released our code at https://github.com/mj-hwang/ReCouPLe

Related papers

• Learning to Hint for Reinforcement Learning [51.46328710610512]
  Group Relative Policy Optimization (GRPO) is widely used for reinforcement learning with verifiable rewards.<n> GRPO often suffers from advantage collapse when all rollouts in a group receive the same reward.<n>We propose Hint Learning for Reinforcement Learning (HiLL), a framework that jointly trains a hinter policy and a reasoner policy.
  arXiv  Detail & Related papers  (2026-04-01T09:58:08Z)
• A Rubric-Supervised Critic from Sparse Real-World Outcomes [87.11204512676193]
  Real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse.<n>We propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling.
  arXiv  Detail & Related papers  (2026-03-04T07:23:54Z)
• Rectifying Shortcut Behaviors in Preference-based Reward Learning [46.09046818725698]
  In reinforcement learning, preference-based reward models play a central role in aligning large language models to human-aligned behavior.<n>Recent studies show that these models are prone to reward hacking and often fail to generalize well due to over-optimization.<n>We introduce a principled yet flexible approach to mitigate shortcut behaviors in preference-based reward learning.
  arXiv  Detail & Related papers  (2025-10-21T20:08:32Z)
• Pref-GUIDE: Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learning [3.513501558121131]
  We propose Pref-GUIDE, a framework that transforms real-time scalar feedback into preference-based data.<n>Pref-GUIDE Individual mitigates temporal inconsistency by comparing agent behaviors within short windows.<n>Pref-GUIDE Voting enhances robustness by aggregating reward models across a population of users to form consensus preferences.
  arXiv  Detail & Related papers  (2025-08-10T00:18:44Z)
• REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and human preferences can lead to catastrophic outcomes in the real world.<n>Recent methods aim to mitigate misalignment by learning reward functions from human preferences.<n>We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• Learning Optimal Advantage from Preferences and Mistaking it for Reward [43.58066500250688]
  Most recent work assumes that human preferences are generated based only upon the reward accrued within those segments, or their partial return. We investigate the consequences of assuming preferences are based upon partial return when they actually arise from regret. This paper overall provides insight regarding why learning under the partial return preference model tends to work so well in practice, despite it conforming poorly to how humans give preferences.
  arXiv  Detail & Related papers  (2023-10-03T21:58:24Z)
• Causal Confusion and Reward Misidentification in Preference-Based Reward Learning [33.944367978407904]
  We study causal confusion and reward misidentification when learning from preferences. We find that the presence of non-causal distractor features, noise in the stated preferences, and partial state observability can all exacerbate reward misidentification.
  arXiv  Detail & Related papers  (2022-04-13T18:41:41Z)
• B-Pref: Benchmarking Preference-Based Reinforcement Learning [84.41494283081326]
  We introduce B-Pref, a benchmark specially designed for preference-based RL. A key challenge with such a benchmark is providing the ability to evaluate candidate algorithms quickly. B-Pref alleviates this by simulating teachers with a wide array of irrationalities.
  arXiv  Detail & Related papers  (2021-11-04T17:32:06Z)
• Reinforcement Based Learning on Classification Task Could Yield Better Generalization and Adversarial Accuracy [0.0]
  We propose a novel method to train deep learning models on an image classification task. We use a reward-based optimization function, similar to the vanilla policy gradient method used in reinforcement learning.
  arXiv  Detail & Related papers  (2020-12-08T11:03:17Z)
• Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting [100.75479161884935]
  We propose a novel training paradigm called Remembering for the Right Reasons (RRR) RRR stores visual model explanations for each example in the buffer and ensures the model has "the right reasons" for its predictions. We demonstrate how RRR can be easily added to any memory or regularization-based approach and results in reduced forgetting.
  arXiv  Detail & Related papers  (2020-10-04T10:05:27Z)
• PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation [58.98802062945709]
  We propose a novel Predicate-Correlation Perception Learning scheme to adaptively seek out appropriate loss weights. Our PCPL framework is further equipped with a graph encoder module to better extract context features.
  arXiv  Detail & Related papers  (2020-09-02T08:30:09Z)
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction [96.90215318875859]
  We show that bootstrapping-based Q-learning algorithms do not necessarily benefit from corrective feedback. We propose a new algorithm, DisCor, which computes an approximation to this optimal distribution and uses it to re-weight the transitions used for training.
  arXiv  Detail & Related papers  (2020-03-16T16:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.