RLVP: Penalize the Path, Reward the Outcome

• URL: http://arxiv.org/abs/2607.07435v1
• Date: Wed, 08 Jul 2026 14:06:14 GMT
• Title: RLVP: Penalize the Path, Reward the Outcome
• Abstract summary: Reinforcement learning from verifiable rewards (RLVR) is blind to both challenges.<n>Attempts to densify supervision by rewarding progress target the hard-to-verify direction.<n>The resulting recipe "penalize the path, reward the outcome" achieves high task success with near-zero violations.
• Score: 0.33451037881913753
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agents acting on our behalf in the real world (e.g. placing phone calls) must learn online from costly, often irreversible interactions rather than cheap simulator steps. Two things follow. First, deployability depends on the path, not only the outcome. An agent must respect outcome-neutral constraints such as not repeatedly calling an unresponsive user, respecting business hours, or completing required authentication constraints that outcome-based rewards cannot express, since violating them frequently improves apparent success. Second, because each interaction is expensive, the agent must learn efficiently from very few examples. Reinforcement learning from verifiable rewards (RLVR) is blind to both challenges: it optimizes solely on the outcome and wastes expensive rollouts on all-fail groups where group-relative advantage collapses to zero. Attempts to densify supervision by rewarding progress target the hard-to-verify direction. In contrast, real agentic environments can cheaply detect bad moves. Since group-relative advantage is equivalent to within-group variance, a dense signal helps only when it supplies variance the outcome lacks. A verifiable penalty on the path meets this condition reliably, while a progress potential helps only where partial progress is reachable. The resulting recipe "penalize the path, reward the outcome" achieves high task success with near-zero violations, where outcome-only training violates constraints on nearly every episode. We provide four design rules for effective penalties, including avoidance of the inaction trap that arises when a penalty is used in isolation.

Related papers

• Tail-Likelihood Reinforcement Learning [63.90013981311359]
  Tail-Likelihood Reinforcement Learning (TailRL) maximizes the log-probability of exceeding a randomly chosen reward threshold.<n>TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines.
  arXiv  Detail & Related papers  (2026-09-02T14:54:45Z)
• FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents [49.519184792774304]
  FailForge is an agentic framework that converts failed rollouts into training signal.<n>FailForge recovers over 26% of previously failed instances at marginal additional cost.
  arXiv  Detail & Related papers  (2026-08-09T08:22:57Z)
• Learning Process Rewards via Success Visitation Matching for Efficient RL [54.67547527674048]
  We propose a simple approach to transform a sparse outcome reward into a dense process reward.<n>Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes.<n>By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion.
  arXiv  Detail & Related papers  (2026-06-22T17:30:24Z)
• Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes [40.20394793623452]
  When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome.<n>We propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for viability and efficiency.<n>HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.
  arXiv  Detail & Related papers  (2026-06-15T17:57:14Z)
• Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short [51.667769734342635]
  Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models.<n>We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them.<n>We show that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks.
  arXiv  Detail & Related papers  (2026-06-08T11:57:17Z)
• Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL [76.45061154544568]
  Self-play reinforcement learning trains language models on their own generated tasks, co-evolving a proposer and solver without human labels.<n>Recent systems report strong reasoning gains, but collapse and instability are widely observed and poorly understood.<n>We argue instead that self-play stability is governed by two distinct levers: a data-level gate that decides which proposer-generated tasks enter the training pool, and the reward signal that updates the policy on tasks already admitted.
  arXiv  Detail & Related papers  (2026-05-21T09:19:23Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• AI Alignment via Incentives and Correction [14.986111703734222]
  We study AI alignment through the lens of law-and-economics models of deterrence and enforcement.<n>A solver may benefit from producing a persuasive but incorrect answer, hiding uncertainty, or exploiting spurious shortcuts.<n>We formalize this interaction in a two-agent model in which a principal chooses rewards over joint correction outcomes.
  arXiv  Detail & Related papers  (2026-05-02T23:28:02Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning [25.562101968892833]
  Reinforcement Learning with Verifiable Rewards (RLVR) elicits long chain-of-thought reasoning in large language models (LLMs)<n>Existing approaches improve RLVR via token-level entropy or sequence-level length control, but lack a semantically grounded, step-level measure of reasoning progress.<n>We propose Step Potential Advantage Estimation (SPAE), a fine-grained credit assignment method that amplifies potential gains, penalizes potential drops, and applies penalty after potential saturates to encourage timely termination.
  arXiv  Detail & Related papers  (2026-01-07T11:36:01Z)
• R$^3$L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification [44.99719889905381]
  Reinforcement learning drives recent advances in reasoning and agentic capabilities, yet current approaches struggle with both exploration and exploitation.<n>We propose R$3$L, Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification.<n>Experiments on agentic and reasoning tasks demonstrate 5% to 52% relative improvements over baselines while maintaining training stability.
  arXiv  Detail & Related papers  (2026-01-07T09:04:52Z)
• Rethinking Reward Miscalibration of GRPO in Agentic RL [18.495499496405635]
  We show that outcome based reward ensures expected negative advantage for those flawed middle steps.<n>We propose training the actor to classify good or bad actions to separate the embedding of good/bad actions.
  arXiv  Detail & Related papers  (2025-09-28T13:24:38Z)
• Semi-supervised reward learning for offline reinforcement learning [71.6909757718301]
  Training agents usually requires reward functions, but rewards are seldom available in practice and their engineering is challenging and laborious. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards.
  arXiv  Detail & Related papers  (2020-12-12T20:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.