Value-Gradient Hypothesis of RL for LLMs

• URL: http://arxiv.org/abs/2605.21654v1
• Date: Wed, 20 May 2026 19:09:05 GMT
• Title: Value-Gradient Hypothesis of RL for LLMs
• Abstract summary: Reinforcement learning substantially improves pretrained language models.<n>It remains understudied why critic-free methods such as PPO and GRPO work as well as they do, and when they should provide the largest gains.
• Score: 1.983259889617852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning substantially improves pretrained language models, but it remains understudied why critic-free methods such as PPO and GRPO work as well as they do, and when they should provide the largest gains. We develop a value-gradient perspective of critic-free RL for LLM post-training. First, under a differentiable rollout and additive-noise parameterization, we show that the actor update is value-gradient-like in expectation: the backward pass propagates costates whose conditional expectation equals the value gradient. Second, for discrete transformer policies, we show that autodifferentiation through attention produces empirical costates that approximate this value signal, with an error controlled by the sampling gap and policy entropy. These results motivate a decomposition of RL impact into value gradient signal and reachable reward headroom, yielding a criterion for when RL should be most effective along a pretraining trajectory.

Related papers

• Start Classifying: Categorical Critics for LLM Reinforcement Learning [35.51572984401546]
  HL-Gauss PPO replaces the scalar MSE head with a categorical predictor over a discretized value support.<n>It consistently improves over strong PPO and DAPO baselines.<n>Controls with one-hot, two-hot, and Bernoulli two-bin critics show that neither a larger output head nor binary classification alone explains the gains.
  arXiv  Detail & Related papers  (2026-08-03T13:05:38Z)
• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training [69.32453275232662]
  We show that a learned critic can inject estimation noise that exceeds the state signal it captures, increasing rather than reducing advantage variance.<n>We propose Explained Variance Policy Optimization (EVPO), which monitors batch-level EV at each training step and adaptively switches between critic-based and batch-mean advantage estimation.
  arXiv  Detail & Related papers  (2026-04-21T14:07:39Z)
• Reinforcement Learning via Value Gradient Flow [32.59594977900539]
  We study behavior-regularized reinforcement learning (RL), where regularization toward a reference distribution is essential to prevent value over-optimization.<n>We propose Value Gradient Flow (VGF), a scalable new paradigm for behavior-regularized RL.<n>VGF casts behavior-regularized RL as an optimal transport problem that maps the reference distribution to the value-induced optimal policy distribution.
  arXiv  Detail & Related papers  (2026-04-15T17:12:56Z)
• Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning [18.371323066068104]
  Credit assignment is a central challenge in reinforcement learning (RL)<n>We propose Generative Actor-Critic (GenAC) that replaces one-shot scalar value prediction with a generative critic.<n>GenAC improves value approximation, ranking reliability, and out-of-distribution generalization.
  arXiv  Detail & Related papers  (2026-04-12T15:54:11Z)
• Off-Policy Value-Based Reinforcement Learning for Large Language Models [25.962820072445222]
  ReVal is a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification.<n>On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO.
  arXiv  Detail & Related papers  (2026-03-24T15:55:02Z)
• REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge [83.2858110368572]
  We propose textbfREAL (underlineREgression-underlineAware Reinforcement underlineLThought), a principled RL framework designed to optimize regression rewards.<n>We show that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods.
  arXiv  Detail & Related papers  (2026-03-17T21:19:08Z)
• A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization [58.116300485427764]
  Reinforcement learning post-training can elicit reasoning behaviors in large language models.<n> token-level correction often leads to unstable training dynamics when the degree of off-policyness is large.<n>We propose a simple yet effective objective, Minimum Prefix Ratio (MinPRO)
  arXiv  Detail & Related papers  (2026-01-30T08:47:19Z)
• Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values [53.72318444646282]
  We propose Reinforcement Learning with Explicit Human Values (RLEV)<n>RLEV aligns Large Language Model (LLM) optimization directly with quantifiable human value signals.<n>We show RLEV consistently outperforms correctness-only baselines across multiple RL algorithms and model scales.
  arXiv  Detail & Related papers  (2025-10-23T04:15:22Z)
• Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance [52.65461207786633]
  Policy-based Reinforcement Learning from Human Feedback is essential for aligning large language models with human preferences.<n>It requires joint training of an actor and critic with a pretrained, fixed reward model for guidance.<n>We propose textbfDecoupled Value Policy Optimization (DVPO), a lean framework that replaces traditional reward modeling with a pretrained emphglobal value model (GVM)
  arXiv  Detail & Related papers  (2025-02-24T08:11:33Z)
• VinePPO: Refining Credit Assignment in RL Training of LLMs [66.80143024475635]
  We propose VinePPO, a straightforward approach that leverages the flexibility of language environments to compute unbiased Monte Carlo-based estimates.<n>Our method consistently outperforms PPO and other baselines across MATH and GSM8K datasets in less wall-clock time.
  arXiv  Detail & Related papers  (2024-10-02T15:49:30Z)
• The Value-Improvement Path: Towards Better Representations for Reinforcement Learning [46.70945548475075]
  We argue that the value prediction problems faced by an RL agent should not be addressed in isolation, but as a single, holistic, prediction problem. An RL algorithm generates a sequence of policies that, at least approximately, improve towards the optimal policy. We demonstrate that a representation that spans the past value-improvement path will also provide an accurate value approximation for future policy improvements.
  arXiv  Detail & Related papers  (2020-06-03T12:51:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.