Mitigating Reward Hacking in RLHF via Advantage Sign Robustness

• URL: http://arxiv.org/abs/2604.02986v1
• Date: Fri, 03 Apr 2026 11:45:16 GMT
• Title: Mitigating Reward Hacking in RLHF via Advantage Sign Robustness
• Abstract summary: We make the assumption that reward hacking is often caused by flipped advantage signs.<n>By considering an adversarial perturbation in the RM parameter space, we can derive a certified sign-preservation radius.<n>We propose Sign-Certified Policy Optimization (SignCert-PO), down-weighting non-robust completions in the policy gradient update.
• Score: 40.66292181512935
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward models (RMs) used in reinforcement learning from human feedback (RLHF) are vulnerable to reward hacking: as the policy maximizes a learned proxy reward, true quality plateaus or degrades. We make the assumption that reward hacking is often caused by flipped advantage signs: instead of reducing the likelihood of a bad response, a flipped sign causes the update to increase it. By considering an adversarial perturbation in the RM parameter space, we can derive a certified sign-preservation radius, which is the smallest perturbation that can flip the advantage sign during policy optimization. Based on this formulation, we propose Sign-Certified Policy Optimization (SignCert-PO), down-weighting non-robust completions in the policy gradient update. Unlike prior approaches that require multiple RMs or access to the RM training data, SignCert-PO is lightweight and operates purely at the policy optimization stage using only the RM parameters and on-policy completions. On TL;DR summarization and AlpacaFarm benchmarks, SignCert-PO consistently achieves a better win rate than baselines and reduces reward hacking.

Related papers

• VIMPO: Value-Implicit Policy Optimization for LLMs [106.88933849641272]
  Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token.<n>Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability.<n>We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from optimality conditions of KL-regularized reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-18T09:44:12Z)
• The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning [54.0056619145783]
  Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback.<n>We identify a hidden bias in PRMs caused by severe imbalance in step-level training data.<n>We propose PRISM, a policy-aware PRM training framework that learns from contrastive step-level comparisons.
  arXiv  Detail & Related papers  (2026-06-08T06:22:33Z)
• The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement [32.29319213544292]
  Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data.<n>We propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training.
  arXiv  Detail & Related papers  (2026-05-29T06:25:16Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking [67.20568716300272]
  Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking.<n>We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models.<n>We show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
  arXiv  Detail & Related papers  (2026-02-23T01:14:53Z)
• Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models [68.45272703833209]
  We show that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure.<n>We introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities.<n>We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment.
  arXiv  Detail & Related papers  (2026-02-20T23:38:03Z)
• RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS [37.07631000345777]
  Policy models can exploit a vanilla Reward Model (RM) by generating acoustic artifacts to achieve spurious rewards, but at the cost of degrading quality.<n>We propose Robust Reward Policy Optimization (RRPO), a novel framework that employs a hybrid regularization scheme.<n>This scheme develops a robust RM whose reward signal is more reliably aligned with human perception, compelling the policy to abandon detrimental shortcuts and instead learn the complex features of genuine emotions.
  arXiv  Detail & Related papers  (2025-12-04T08:12:49Z)
• Repairing Reward Functions with Human Feedback to Mitigate Reward Hacking [13.417125511014447]
  We propose an automated framework that repairs a human-specified proxy reward function by learning an additive, transition-dependent correction term from preferences.<n>PBRR consistently outperforms baselines that learn a reward function from scratch from preferences or modify the proxy reward function using other approaches.
  arXiv  Detail & Related papers  (2025-10-14T23:18:24Z)
• Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback [52.1410307583181]
  We useReinforcement Learning from Human Feedback to train language models (LMs) to follow complex human preferences.<n>As training progresses, the responses generated by the LM no longer resemble the responses seen by the reward model (RM)<n>We propose Off-Policy Corrected Reward Modeling to correct the RM using importance weighting, without requiring new labels or samples.
  arXiv  Detail & Related papers  (2025-07-21T11:19:04Z)
• Adversarial Training of Reward Models [74.17196154247964]
  We introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples.<n>By leveraging reinforcement learning, Adv-RM trains a policy to expose vulnerabilities in large state-of-the-art reward models.<n>We demonstrate that Adv-RM significantly outperforms conventional reward training.
  arXiv  Detail & Related papers  (2025-04-08T15:38:25Z)
• Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking [11.589217788048964]
  We introduce a definition of reward hacking based on the correlation between proxy and true rewards for states.<n>We show theoretically that regularization to the reference policy can effectively prevent reward hacking.
  arXiv  Detail & Related papers  (2024-03-05T18:22:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.