Learning Process Rewards via Success Visitation Matching for Efficient RL

• URL: http://arxiv.org/abs/2606.23640v1
• Date: Mon, 22 Jun 2026 17:30:24 GMT
• Title: Learning Process Rewards via Success Visitation Matching for Efficient RL
• Abstract summary: We propose a simple approach to transform a sparse outcome reward into a dense process reward.<n>Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes.<n>By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion.
• Score: 54.67547527674048
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In many modern applications of reinforcement learning (RL), the natural reward for a task of interest is inherently sparse: a reward of 0 is given everywhere except when the task is completed, when a reward of +1 is given. Training a policy to maximize such a sparse reward requires solving a challenging credit assignment problem, leading to slow or ineffective RL improvement. We propose a simple approach to transform a sparse outcome reward into a dense process reward. Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes, and using this discriminator to incentivize the RL-learned policy to match the state-action visitations of successful episodes, while avoiding those of unsuccessful episodes. By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion, and, we show, provably achieves this without changing the optimal policy. Focusing on finetuning of robotic control policies, we demonstrate that our approach leads to significantly faster RL finetuning performance on both simulated and real-world manipulation tasks, as compared to simply maximizing the sparse outcome reward.

Related papers

• Match or Replay: Self Imitating Proximal Policy Optimization [14.033176618356746]
  Reinforcement Learning (RL) agents often struggle with inefficient exploration.<n>Traditional exploration strategies can lead to slow learning and suboptimal performance.<n>We propose a self-imitating on-policy algorithm that enhances exploration and sample efficiency.
  arXiv  Detail & Related papers  (2026-03-29T04:44:48Z)
• Reward-Conditioned Reinforcement Learning [56.417273471201845]
  We introduce Reward-Conditioned Reinforcement Learning (RCRL), a framework that trains a single agent to optimize a family of reward specifications.<n>RCRL conditions the agent on reward parameterizations and learns multiple reward objectives from a shared replay data entirely off-policy.<n>Our results demonstrate that RCRL provides a scalable mechanism for learning robust, steerable policies without sacrificing the simplicity of single-task training.
  arXiv  Detail & Related papers  (2026-03-05T11:29:17Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution [9.181156720071547]
  Reinforcement learning holds promise for training agents to handle complex, goal-oriented tasks.<n> Feedback signals are typically available only after the entire task is completed.<n>We propose Stepwise Progress Attribution, which decomposes the final reward into stepwise contributions.
  arXiv  Detail & Related papers  (2025-05-27T05:21:04Z)
• Process Reinforcement through Implicit Rewards [94.09453548052862]
  Dense process rewards have proven a more effective alternative to the sparse outcome-level rewards in the inference-time scaling of large language models (LLMs)<n>Dense rewards also offer an appealing choice for the reinforcement learning (RL) of LLMs since their fine-grained rewards have the potential to address some inherent issues of outcome rewards.<n>This can be primarily attributed to the challenges of training process reward models (PRMs) online, where collecting high-quality process labels is prohibitively expensive.<n>We propose PRIME, which enables online PRM updates using only policy rollouts and outcome labels through implict process rewards
  arXiv  Detail & Related papers  (2025-02-03T15:43:48Z)
• Adaptive Reward Design for Reinforcement Learning [2.3031174164121127]
  We propose a suite of reward functions that incentivize an RL agent to complete a task specified by a formula as much as possible.<n>We develop an adaptive reward shaping approach that dynamically updates reward functions during the learning process.
  arXiv  Detail & Related papers  (2024-12-14T18:04:18Z)
• Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Control [9.118706387430883]
  Reinforcement learning (RL) has recently proven great success in various domains. Yet, the design of the reward function requires detailed domain expertise and tedious fine-tuning to ensure that agents are able to learn the desired behaviour. We propose to use model predictive control(MPC) as an experience source for training RL agents in sparse reward environments.
  arXiv  Detail & Related papers  (2022-10-04T11:06:38Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RL [91.26538493552817]
  We present a formulation of hindsight relabeling for meta-RL, which relabels experience during meta-training to enable learning to learn entirely using sparse reward. We demonstrate the effectiveness of our approach on a suite of challenging sparse reward goal-reaching environments.
  arXiv  Detail & Related papers  (2021-12-02T00:51:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.