Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning

• URL: http://arxiv.org/abs/2604.10701v1
• Date: Sun, 12 Apr 2026 15:54:11 GMT
• Title: Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning
• Abstract summary: Credit assignment is a central challenge in reinforcement learning (RL)<n>We propose Generative Actor-Critic (GenAC) that replaces one-shot scalar value prediction with a generative critic.<n>GenAC improves value approximation, ranking reliability, and out-of-distribution generalization.
• Score: 18.371323066068104
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Credit assignment is a central challenge in reinforcement learning (RL). Classical actor-critic methods address this challenge through fine-grained advantage estimation based on a learned value function. However, learned value models are often avoided in modern large language model (LLM) RL because conventional discriminative critics are difficult to train reliably. We revisit value modeling and argue that this difficulty is partly due to limited expressiveness. In particular, representation complexity theory suggests that value functions can be hard to approximate under the one-shot prediction paradigm used by existing value models, and our scaling experiments show that such critics do not improve reliably with scale. Motivated by this observation, we propose Generative Actor-Critic (GenAC), which replaces one-shot scalar value prediction with a generative critic that performs chain-of-thought reasoning before producing a value estimate. We further introduce In-Context Conditioning, which helps the critic remain calibrated to the current actor throughout training. GenAC improves value approximation, ranking reliability, and out-of-distribution generalization, and these gains translate into stronger downstream RL performance than both value-based and value-free baselines. Overall, our results suggest that stronger value modeling is a promising direction for improving credit assignment in LLM reinforcement learning.

Related papers

• Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning [18.614878630206487]
  Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals.<n>We introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks.<n>We propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM's hidden states as representation to weighted average disjoint rollouts and their return.
  arXiv  Detail & Related papers  (2026-05-28T11:31:13Z)
• Value-Gradient Hypothesis of RL for LLMs [1.983259889617852]
  Reinforcement learning substantially improves pretrained language models.<n>It remains understudied why critic-free methods such as PPO and GRPO work as well as they do, and when they should provide the largest gains.
  arXiv  Detail & Related papers  (2026-05-20T19:09:05Z)
• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards [73.44333771806282]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision.<n>CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors.
  arXiv  Detail & Related papers  (2026-05-14T08:22:21Z)
• On Predicting the Post-training Potential of Pre-trained LLMs [60.07459271263409]
  We introduce a new task of predicting post-training potential - forecasting a base model's performance before post-training.<n>We propose RuDE, a unified framework that bypasses the generation gap of base models by leveraging response discrimination.<n>Experiments demonstrate a correlation greater than 90% with post-training performance.
  arXiv  Detail & Related papers  (2026-05-12T11:33:49Z)
• Context-Value-Action Architecture for Value-Driven Large Language Model Agents [15.830984215635008]
  Large Language Models (LLMs) have shown promise in simulating human behavior, yet existing agents often exhibit behavioral rigidity.<n>We propose the Context-Value-Action architecture, grounded in the Stimulus-Organism-Response (S-O-R) model and Schwartz's Theory of Basic Human Values.<n>Our approach effectively mitigates polarization while offering superior behavioral fidelity and interpretability.
  arXiv  Detail & Related papers  (2026-04-07T14:34:20Z)
• $V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts [81.48669089692189]
  Generalist Value Models (such as $V_0.5$) achieve pre-trained value estimation by explicitly encoding model capabilities in-context.<n>We propose $V_0.5$, which adaptively fuses the baseline predicted by such value model with the empirical mean derived from sparse rollouts.<n>$V_0.5$ significantly outperforms GRPO and DAPO, achieving faster convergence and over some 10% performance improvement.
  arXiv  Detail & Related papers  (2026-03-11T14:57:41Z)
• Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling [49.41422138354821]
  We propose a principled reward modeling framework that integrates non-negative factor analysis into the Bradley-Terry preference model.<n>BNRM represents rewards through a sparse, non-negative latent factor generative process.<n>We show that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.
  arXiv  Detail & Related papers  (2026-02-11T08:14:11Z)
• LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model [99.71684530652942]
  We show that LLaVA-Critic-R1 emerges as a top-performing critic but also as a competitive policy model.<n>Applying self-critique at test time yields an average +13.8% improvement on five representative reasoning tasks.<n>Our results reveal that RL training on critic data can produce a unified model excelling at both evaluation and generation.
  arXiv  Detail & Related papers  (2025-08-31T03:08:02Z)
• Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws [52.10468229008941]
  This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting.<n>We provide theoretical insights into why this approach improves generalization and data efficiency compared to training without a reference model.<n>Building on these insights, we introduce a novel method for Contrastive Language-Image Pretraining with a reference model, termed DRRho-CLIP.
  arXiv  Detail & Related papers  (2025-05-10T16:55:03Z)
• Teaching Language Models to Critique via Reinforcement Learning [59.36253627145115]
  We show that critics trained with $textttCTRL$ significantly enhance pass rates and mitigate errors across both base and stronger generator models.<n>We also show that these critic models act as accurate generative reward models and enable test-time scaling through iterative critique-revision.
  arXiv  Detail & Related papers  (2025-02-05T02:18:46Z)
• Rethinking and Recomputing the Value of Machine Learning Models [16.06614967567121]
  We argue that the prevailing approach to training and evaluating machine learning models often fails to consider their real-world application.<n>Traditional metrics like accuracy and f-score fail to capture the beneficial value of models in such hybrid settings.<n>We introduce a simple yet theoretically sound "value" metric that incorporates task-specific costs for correct predictions, errors, and rejections.
  arXiv  Detail & Related papers  (2022-09-30T01:02:31Z)
• Value Gradient weighted Model-Based Reinforcement Learning [28.366157882991565]
  Model-based reinforcement learning (MBRL) is a sample efficient technique to obtain control policies. VaGraM is a novel method for value-aware model learning.
  arXiv  Detail & Related papers  (2022-04-04T13:28:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.