Measuring Reward-Seeking via Contrastive Belief Updates

• URL: http://arxiv.org/abs/2607.18966v1
• Date: Tue, 21 Jul 2026 10:57:09 GMT
• Title: Measuring Reward-Seeking via Contrastive Belief Updates
• Abstract summary: Language models may learn to optimize the grader's judgment rather than the intended objective.<n>We measure reward-seeking using Contrastive Synthetic Document Finetuning.<n>A model organism trained to reward-hack (gpt-oss-120b) is more than twice as sensitive to grader preferences as the unmodified model.
• Score: 35.6734895760596
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models trained with reinforcement learning may learn to optimize the grader's judgment rather than the intended objective. This "reward-seeking" is difficult to measure because a model that pursues the grader's judgment and one that pursues the intended objective behave identically whenever the grader rewards the intended behavior. We measure reward-seeking using Contrastive Synthetic Document Finetuning to change a model's beliefs about what the grader rewards, putting those beliefs in conflict with what users or developers want, and measuring the rate at which the model adopts each party's preferred behavior. Applied to intermediate checkpoints of a capabilities-focused OpenAI o3 RL run, without safety training, we find that these checkpoints often side with grader preferences over those of users or developers on coding and alignment tasks. This tendency to side with the grader trends upward throughout RL training. For example, in an environment that forces a choice between keeping a promise to a supervisor and breaking it to complete the task, a late capabilities-focused o3 checkpoint breaks the promise 87% of the time when SDF documents say the grader rewards task completion, versus 9% when they say it rewards honesty (a choice its chain-of-thought often makes explicit). An earlier checkpoint is far less sensitive (40% vs. 24%). Our method also generalizes to reward-hacking models. A model organism trained to reward-hack (gpt-oss-120b) is more than twice as sensitive to grader preferences as the unmodified model, with the mean behavioral shift in favor of the grader rising from 33% to 86%. These results indicate that RL can increase reward-seeking over the course of training, producing models that may act against their developers' intentions when they believe that doing so leads to higher reward.

Related papers

• Discretizing Reward Models [70.71071807050916]
  We show that many popular reward models are oversensitive, assigning different scores to equally good responses.<n>We propose evaluating reward models using measures of "discriminative ability" and "specificity"
  arXiv  Detail & Related papers  (2026-06-19T23:13:59Z)
• Understanding and Mitigating Premature Confidence for Better LLM Reasoning [76.16007941549857]
  Long chains of thought (CoT) from current language models frequently contain logical gaps and unjustified leaps.<n>We find such a signal in how the model's confidence evolves during reasoning.<n>We exploit this in progressive confidence shaping, a reinforcement learning objective that trains models to update their confidence as they reason rather than commit early.
  arXiv  Detail & Related papers  (2026-05-23T04:42:45Z)
• Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation [13.633897375946129]
  We study pass-rate rewards in critic-free RL for code generation.<n>We find that pass-rate rewards are denser, but the induced gradient updates do not consistently move probability mass toward full-pass solutions.
  arXiv  Detail & Related papers  (2026-05-01T10:26:43Z)
• When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient [28.276272279321578]
  Training language models via reinforcement learning often relies on imperfect proxy rewards.<n>Standard metrics for assessing the quality of proxy rewards, such as ranking accuracy, treat incorrect rewards as strictly harmful.<n>We categorize reward errors according to their effect on the increase in ground truth reward.
  arXiv  Detail & Related papers  (2026-04-28T17:10:15Z)
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking [67.20568716300272]
  Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking.<n>We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models.<n>We show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
  arXiv  Detail & Related papers  (2026-02-23T01:14:53Z)
• Stabilizing Long-term Multi-turn Reinforcement Learning with Gated Rewards [13.70228195630989]
  We introduce the SWE-oriented RL Framework, a unified system supporting multi-turn interaction, docker-based execution, and customizable reward functions.<n>We also propose Gated Reward Accumulation (G-RA), a novel method that accumulates immediate rewards only when high-level (long-term) rewards meet a predefined threshold.
  arXiv  Detail & Related papers  (2025-08-14T11:37:02Z)
• Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning [8.677768413982802]
  Language models can engage in reward hacking without revealing this behavior in their chain-of-thought reasoning.<n>We propose verbalization fine-tuning (VFT) to train models to explicitly acknowledge when they are influenced by prompt cues.<n>Our results show that teaching models to explicitly verbalize reward hacking behavior before RL significantly improves their detection.
  arXiv  Detail & Related papers  (2025-06-28T06:37:10Z)
• What Makes a Reward Model a Good Teacher? An Optimization Perspective [82.73297593767181]
  We prove that regardless of how accurate a reward model is, if it induces low reward variance, then the RLHF objective suffers from a flat landscape.<n>We additionally show that a reward model that works well for one language model can induce low reward variance, and thus a flat objective landscape, for another.
  arXiv  Detail & Related papers  (2025-03-19T17:54:41Z)
• RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution [50.171320156632866]
  Reinforcement learning from human feedback offers a promising approach to aligning large language models with human preferences.<n>Current reward models operate as sequence-to-one models, allocating a single, sparse, and delayed reward to an entire output sequence.<n>We propose a more fine-grained, token-level guidance approach for RL training.
  arXiv  Detail & Related papers  (2024-11-13T02:45:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.