Stage-Transition Dense Reward Modeling for Reinforcement Learning

• URL: http://arxiv.org/abs/2606.31377v1
• Date: Tue, 30 Jun 2026 09:07:19 GMT
• Title: Stage-Transition Dense Reward Modeling for Reinforcement Learning
• Abstract summary: Reinforcement learning for long-horizon robotic manipulation is often limited by sparse and delayed rewards.<n>This work proposes Stage-Transition Dense Reward (STDR), a visual reward-learning framework that converts unstructured expert videos into logically grounded dense rewards for training RL agents from scratch.<n>STDR delivers two complementary learning signals during online training: (i) stage-transition feedback that provides goal-directed reward, and (ii) within-stage progress feedback that supplies fine-grained guidance toward completing each stage.
• Score: 22.634812963829194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning for long-horizon robotic manipulation is often limited by sparse and delayed rewards, while manually designing dense shaping signals is costly and brittle to changes in environments and object configurations. This work proposes Stage-Transition Dense Reward (STDR), a visual reward-learning framework that converts unstructured expert videos into logically grounded dense rewards for training RL agents from scratch. STDR leverages semantic understanding to infer a task's stage structure from demonstrations, and delivers two complementary learning signals during online training: (i) stage-transition feedback that provides goal-directed reward, and (ii) within-stage progress feedback that supplies fine-grained guidance toward completing each stage. Furthermore, an out-of-distribution (OOD) detection mechanism and a grasping regulation module are integrated to enhance robustness and prevent reward hacking. Experiments on 14 manipulation tasks across MetaWorld, ManiSkill, and Franka Kitchen show that STDR consistently improves sample efficiency and success rates over multiple baselines, and matches or surpasses handcrafted dense rewards on several challenging tasks. Real-robot evaluations further indicate that STDR assigns stable, progress-aligned rewards on successful executions while producing appropriately low rewards for failures, suggesting robustness to visual noise and better-calibrated reward assignment across settings.

Related papers

• Iron: Intent-Aligned and Retrospective Dual Learning Framework for Enhancing Generalist Virtual Agents [31.811763840603614]
  We introduce Iron, an intent-aligned framework for training GUI agents.<n>Iron employs a stepwise cycle-consistent reward to achieve fine-grained alignment between low-level actions and high-level intents.<n>Experiments demonstrate that Iron-trained generalist agents consistently improve performance on cross-environment and cross-device tasks.
  arXiv  Detail & Related papers  (2026-08-28T03:15:11Z)
• DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation [67.08835970838996]
  Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning.<n>Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels.<n>We introduce DenseReward, a dense robotic reward model that addresses both challenges.
  arXiv  Detail & Related papers  (2026-07-14T17:59:29Z)
• RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents [24.215311212134882]
  We propose Reward-Swap Policy Optimization (RSPO) to leverage the rich information from dense process rewards to facilitate training with outcome rewards.<n>RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards.
  arXiv  Detail & Related papers  (2026-07-06T06:32:39Z)
• RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation [12.756156905476503]
  Reinforcement learning for robot manipulation is often bottlenecked by reward design.<n>We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward.<n>RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding.
  arXiv  Detail & Related papers  (2026-06-20T13:03:21Z)
• ProgAgent:A Continual RL Agent with Progress-Aware Rewards [0.07646713951724009]
  ProgAgent is a continual reinforcement learning agent that unifies progress-aware reward learning with a JAX-native system architecture.<n>It derives dense, shaped rewards from unlabeled expert videos through a perceptual model that estimates task progress across initial, current, and goal observations.<n>ProgAgent supports massively parallel rollouts and fully differentiable updates, rendering a sophisticated unified objective feasible.
  arXiv  Detail & Related papers  (2026-03-08T19:58:07Z)
• Adaptive Milestone Reward for GUI Agents [38.548364518806046]
  We propose the Adaptive Milestone Reward (ADMIRE) mechanism.<n>ADMIRE constructs a verifiable, adaptive reward system by anchoring trajectory to milestones.<n>Experiments demonstrate that ADMIRE consistently yields over 10% absolute improvement in success rate.
  arXiv  Detail & Related papers  (2026-02-12T03:31:40Z)
• Benchmarking Multimodal CoT Reward Model Stepwise by Visual Program [96.79600297158271]
  We propose SVIP, a novel approach to train a step-level multi-dimensional Chain-of-Thought(CoT) reward model automatically.<n>It generates code for solving visual tasks and transforms the analysis of code blocks into the evaluation of CoT step as training samples.<n>We show that SVIP-Reward improves MLLM performance across training and inference-time scaling.
  arXiv  Detail & Related papers  (2025-04-09T06:09:40Z)
• Subtask-Aware Visual Reward Learning from Segmented Demonstrations [97.80917991633248]
  This paper introduces REDS: REward learning from Demonstration with Demonstrations, a novel reward learning framework.<n>We train a dense reward function conditioned on video segments and their corresponding subtasks to ensure alignment with ground-truth reward signals.<n>Our experiments show that REDS significantly outperforms baseline methods on complex robotic manipulation tasks in Meta-World.
  arXiv  Detail & Related papers  (2025-02-28T01:25:37Z)
• Curriculum Reinforcement Learning for Complex Reward Functions [5.78463306498655]
  We propose a two-stage reward curriculum that first maximizes a simple reward function and then transitions to the full, complex reward.<n>We evaluate our method on the DeepMind control suite, modified to include an additional constraint term in the reward definitions.<n>Our results demonstrate the potential of two-stage reward curricula for efficient and stable RL in environments with complex rewards.
  arXiv  Detail & Related papers  (2024-10-22T08:07:44Z)
• Affordance-Guided Reinforcement Learning via Visual Prompting [51.361977466993345]
  Keypoint-based Affordance Guidance for Improvements (KAGI) is a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL.<n>On diverse real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 30K online fine-tuning steps.
  arXiv  Detail & Related papers  (2024-07-14T21:41:29Z)
• RILe: Reinforced Imitation Learning [60.63173816209543]
  RILe (Reinforced Learning) is a framework that combines the strengths of imitation learning and inverse reinforcement learning to learn a dense reward function efficiently.<n>Our framework produces high-performing policies in high-dimensional tasks where direct imitation fails to replicate complex behaviors.
  arXiv  Detail & Related papers  (2024-06-12T17:56:31Z)
• Towards Sequence-Level Training for Visual Tracking [60.95799261482857]
  This work introduces a sequence-level training strategy for visual tracking based on reinforcement learning. Four representative tracking models, SiamRPN++, SiamAttn, TransT, and TrDiMP, consistently improve by incorporating the proposed methods in training.
  arXiv  Detail & Related papers  (2022-08-11T13:15:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.