Verifiable Process Rewards for Agentic Reasoning

• URL: http://arxiv.org/abs/2605.10325v1
• Date: Mon, 11 May 2026 10:30:53 GMT
• Title: Verifiable Process Rewards for Agentic Reasoning
• Abstract summary: Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of large language models (LLMs)<n>Most existing approaches rely on sparse outcome-level feedback.<n>We propose Verifiable Process Rewards (VPR), a framework that converts such oracles into dense turn-level supervision for reinforcement learning.
• Score: 21.195739597726142
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of large language models (LLMs), but most existing approaches rely on sparse outcome-level feedback. This sparsity creates a credit assignment challenge in long-horizon agentic reasoning: a trajectory may fail despite containing many correct intermediate decisions, or succeed despite containing flawed ones. In this work, we study a class of densely-verifiable agentic reasoning problems, where intermediate actions can be objectively checked by symbolic or algorithmic oracles. We propose Verifiable Process Rewards (VPR), a framework that converts such oracles into dense turn-level supervision for reinforcement learning, and instantiate it in three representative settings: search-based verification for dynamic deduction, constraint-based verification for logical reasoning, and posterior-based verification for probabilistic inference. We further provide a theoretical analysis showing that dense verifier-grounded rewards can improve long-horizon credit assignment by providing more localized learning signals, with the benefit depending on the reliability of the verifier. Empirically, VPR outperforms outcome-level reward and rollout-based process reward baselines across controlled environments, and more importantly, transfers to both general and agentic reasoning benchmarks, suggesting that verifiable process supervision can foster general reasoning skills applicable beyond the training environments. Our results indicate that VPR is a promising approach for enhancing LLM agents whenever reliable intermediate verification is available, while also highlighting its dependence on oracle quality and the open challenge of extending VPR to less structured, open-ended environments.

Related papers

• Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agents [23.69429098915734]
  Verifier-Guided Action Selection (VegAS) is a test-time framework designed to improve robustness of MLLM-based embodied agents.<n>We show that VeGAS consistently improves generalization, achieving up to a 36% relative performance gain over strong CoT baselines.
  arXiv  Detail & Related papers  (2026-05-12T18:08:24Z)
• Less Approximates More: Harmonizing Performance and Confidence Faithfulness via Hybrid Post-Training for High-Stakes Tasks [52.81286869496811]
  Large language models are increasingly deployed in high-stakes tasks, where confident yet incorrect inferences may cause real-world harm.<n>We propose to jointly optimize unsupervised Reinforcement Learning from Internal Feedback with reasoning-trace-guided Reasoning Distillation.<n>We introduce HyTuning, a hybrid post-training framework that adaptively reweights RD and RLIF via a PRG-style metric.
  arXiv  Detail & Related papers  (2026-04-09T16:50:11Z)
• Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration [49.9937230730202]
  We propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention.<n>Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories.<n>We show that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales.
  arXiv  Detail & Related papers  (2026-02-03T15:32:09Z)
• Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning [14.632557283678898]
  Verifiable Process Reward Models (VPRMs) are a reinforcement-learning framework in which intermediate reasoning steps are checked by deterministic, rule-based verifiers.<n>We apply VPRMs to risk-of-bias assessment for medical evidence synthesis.<n>Results show that VPRMs achieve up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards.
  arXiv  Detail & Related papers  (2026-01-23T23:22:20Z)
• Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoning [52.144281362465996]
  We propose EAPO (Evidence-Augmented Policy Optimization) to apply Reinforcement Learning to long-context scenarios.<n>We first establish the Evidence-Augmented Reasoning paradigm, validating via Tree-Structured Evidence Sampling.<n>We then introduce a specialized RL algorithm where a reward model computes a Group-Relative Evidence Reward.<n>To sustain accurate supervision throughout training, we further incorporate an Adaptive Reward-Policy Co-Evolution mechanism.
  arXiv  Detail & Related papers  (2026-01-15T11:40:57Z)
• ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning [17.98065634130798]
  We propose the Intrinsic Confidence-Driven Group Relative Preference Optimization method (ICPO)<n>ICPO calculates a preference advantage score for each response by comparing the relative generation probabilities of multiple responses under the same input prompt.<n>We have discovered that the preference advantage score not only alleviates the issues of coarse-grained rewards and reward noise but also effectively curbs overconfident errors.
  arXiv  Detail & Related papers  (2025-11-26T03:10:15Z)
• From <Answer> to <Think>: Multidimensional Supervision of Reasoning Process for LLM Optimization [62.07990937720985]
  Dimension-level Reward Model (DRM) is a new supervision framework for Large Language Models.<n>DRM evaluates the quality of a reasoning process along three fundamental, complementary, and interpretable dimensions.<n> Experimental results show that DRM provides effective supervision signals, guides the optimization of LLMs and enhances their reasoning ability.
  arXiv  Detail & Related papers  (2025-10-13T14:29:15Z)
• The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives [8.030821324147515]
  Inverse Reinforcement Learning can infer reward functions from behaviour.<n>Existing approaches either produce a single, overconfident reward estimate or fail to address the fundamental ambiguity of the task.<n>This paper introduces a principled auditing framework that re-frames reward inference from a simple estimation task to a comprehensive process for verification.
  arXiv  Detail & Related papers  (2025-10-07T16:25:14Z)
• Veri-R1: Toward Precise and Faithful Claim Verification via Online Reinforcement Learning [53.05161493434908]
  Claim verification with large language models (LLMs) has recently attracted growing attention, due to their strong reasoning capabilities and transparent verification processes.<n>We introduce Veri-R1, an online reinforcement learning framework that enables an LLM to interact with a search engine and to receive reward signals that explicitly shape its planning, retrieval, and reasoning behaviors.<n> Empirical results show that Veri-R1 improves joint accuracy by up to 30% and doubles the evidence score, often surpassing its larger-scale model counterparts.
  arXiv  Detail & Related papers  (2025-10-02T11:49:48Z)
• Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Tasks [12.31210445905605]
  We introduce Principle Process Reward (PPR), an RL approach that unifies step-level assessment and outcome verification.<n>PPR achieves state-of-the-art performance across a wide range of benchmarks, demonstrating its impressive robustness and generalization.
  arXiv  Detail & Related papers  (2025-09-29T23:44:55Z)
• Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning [87.7836502955847]
  We propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning.<n>Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood.<n>We introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy.
  arXiv  Detail & Related papers  (2025-06-10T12:40:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.