Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards

• URL: http://arxiv.org/abs/2609.03342v1
• Date: Thu, 03 Sep 2026 04:00:28 GMT
• Title: Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
• Abstract summary: Gradient Reward (GAR) operates in the policy's own gradient space.<n>C cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward.
• Score: 24.636191382674184
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning from verifiable rewards (RLVR) drives chain-of-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation. We propose Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock overhead. We prove that this cosine admits a multiplicative decomposition into prediction-error and activation-pattern factors, providing a concrete characterization of what the alignment signal measures. On Qwen3-4B and Qwen3-8B, GAR consistently improves over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific data. Code and data are available at https://github.com/LQgdwind/GAR.

Related papers

• Latent Reward Registers for Diffusion Preference Alignment [46.49072106092868]
  We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents.<n>This independent readout mechanism extracts latent reward evidence without altering the generator's hidden states or velocity field.<n>For training, Reward-Gradient On-Policy Distillation (RG-OPD) distills reward-guided updates along on-policy trajectories, bypassing the computationally expensive rollouts of standard policy gradients.<n>For inference, Reward-Guided Sampling (RGS) steers trajectories via magnitude-matched reward gradients without parameter updates.
  arXiv  Detail & Related papers  (2026-08-04T17:00:52Z)
• Geometric Gradient Rectification for Safe Open-Set Semi-Supervised Learning [15.386002863432678]
  Open-set semi-supervised learning aims to leverage unlabeled data that may contain out-of-distribution outliers.<n>Existing methods mainly follow two paradigms: filtering suspicious samples or incorporating unlabeled objectives with soft weighting.<n>We argue that both face a common trade-off: aggressive filtering can discard informative but hard ID samples, whereas utilization can introduce auxiliary gradients that conflict with supervised learning when pseudo labels are wrong.<n>We propose textitGeometric Gradient Rectification (GGR), a plug-in framework that uses the supervised gradient as an anchor and projects conflicting auxiliary gradients onto an admissible
  arXiv  Detail & Related papers  (2026-06-25T12:45:42Z)
• Gradient-Guided Reward Optimization for Inference-time Alignment [22.28081683932275]
  We introduce a lightweight inference-time method that performs targeted, minimal intervention during decoding via gradient guidance.<n>Experiments show that GGRO consistently improves inference-time alignment across safety, helpfulness, and reasoning benchmarks.<n>It also increases coverage of high-quality responses and robustness to reward hacking, with minimal computational overhead.
  arXiv  Detail & Related papers  (2026-06-08T15:33:13Z)
• TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Composition [62.56752617853322]
  Outcome-based rewards provide only sparse feedback, while trajectory-supervised rewards depend on annotated reference solutions.<n>We propose TIER: Trajectory-Invariant Execution Rewards, a reward framework that derives supervision directly from function schemas and runtime execution.
  arXiv  Detail & Related papers  (2026-05-16T03:47:26Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• Beyond Verifiable Rewards: Rubric-Based GRM for Reinforced Fine-Tuning SWE Agents [9.505140329883762]
  We introduce a Generative Reward Model (GRM) that provides richer learning signals.<n>The GRM is equipped with human-designed rubrics that indicate criteria for encouraging or discouraging specific behavioral patterns.<n>When used for Reinforced Fine-Tuning (RFT) on SWE Tasks, our approach outperforms terminal-score-only rejection sampling.
  arXiv  Detail & Related papers  (2026-03-13T02:23:49Z)
• GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning [55.03441672267886]
  We propose GradAlign, a gradient-aligned data selection method for reinforcement learning.<n>We evaluate GradAlign across three data regimes: unreliable reward signals, distribution imbalance, and low-utility training corpus.
  arXiv  Detail & Related papers  (2026-02-25T01:54:50Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.