Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models

• URL: http://arxiv.org/abs/2605.25189v1
• Date: Sun, 24 May 2026 17:34:34 GMT
• Title: Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models
• Abstract summary: Reward hacking arises when a model improves a proxy reward by exploiting shortcuts rather than solving the intended task.<n>We analyze this drift through dominant singular directions of parameter updates and show that reward-hacking runs exhibit substantially larger directional change than clean runs.
• Score: 59.427161616512855
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reward hacking arises when a model improves a proxy reward by exploiting shortcuts rather than solving the intended task. We study this failure mode through the geometry of reinforcement learning updates in language models and argue that hacking emerges when optimization drifts away from a stable low-dimensional learning trajectory. We analyze this drift through dominant singular directions of parameter updates and show that reward-hacking runs exhibit substantially larger directional change than clean runs. Motivated by this observation, we introduce trusted-direction projection, which constrains gradients to remain within a clean reference subspace. Across reward-hacking experiments on mathematical reasoning, the proposed approach delays shortcut exploitation and better preserves task performance.

Related papers

• Reinforcement Learning from Rich Feedback with Distributional DAgger [9.188497824291854]
  Reasoning models have advanced rapidly, but the dominant reinforcement learning from verifiable rewards (RLVR) recipe remains surprisingly narrow.<n>We study how to use such feedback through a distributional variant of the classic imitation learning algorithm DAgger.<n>This yields a simple forward cross-entropy objective that admits a blackbox expert and whose sequence-level gradient conduct rich credit assignment by propagating future expert-student disagreement.
  arXiv  Detail & Related papers  (2026-06-03T17:54:04Z)
• STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning [80.78140312980484]
  We propose language-driven stepwise trajectory redirection, termed as STRIDE.<n>We co-train a generator and a generative verifier using only outcome-based rewards, eliminating external annotations.<n> Experiments on diverse reasoning benchmarks show that STRIDE significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-13T11:04:31Z)
• Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping [66.25536973294726]
  Post-training methods for text-to-image (T2I) models are prone to reward hacking.<n>Super-Linear Advantage Shaping (SLAS) reshapes the local policy space.<n>SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T17:59:25Z)
• When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals [11.280037154530847]
  Reinforcement learning for LLMs is vulnerable to reward hacking.<n>We study this phenomenon in coding tasks using an environment-manipulation setting.<n>We propose Advantage Modification, which integrates shortcut concept scores into GRPO advantage computation.
  arXiv  Detail & Related papers  (2026-04-01T23:33:08Z)
• Rectifying Shortcut Behaviors in Preference-based Reward Learning [46.09046818725698]
  In reinforcement learning, preference-based reward models play a central role in aligning large language models to human-aligned behavior.<n>Recent studies show that these models are prone to reward hacking and often fail to generalize well due to over-optimization.<n>We introduce a principled yet flexible approach to mitigate shortcut behaviors in preference-based reward learning.
  arXiv  Detail & Related papers  (2025-10-21T20:08:32Z)
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning [50.20267980386502]
  We learn a dense, token-level reward model for process supervision directly from expert demonstrations.<n>The learned reasoning reward serves two complementary roles: (i) it provides step-level feedback to optimise a reasoning policy during training; and (ii) it functions at inference as a critic to rerank sampled traces under fixed compute budgets.
  arXiv  Detail & Related papers  (2025-10-02T09:55:26Z)
• Mitigating Backdoor Attacks using Activation-Guided Model Editing [8.00994004466919]
  Backdoor attacks compromise the integrity and reliability of machine learning models. We propose a novel backdoor mitigation approach via machine unlearning to counter such backdoor attacks.
  arXiv  Detail & Related papers  (2024-07-10T13:43:47Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.