Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation

• URL: http://arxiv.org/abs/2608.18787v1
• Date: Wed, 19 Aug 2026 10:44:20 GMT
• Title: Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation
• Abstract summary: We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations.<n>It penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress.<n>Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance.
• Score: 51.43645327777772
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning robotic policies requires dense rewards that remain informative when behavior departs from successful demonstrations. Progress-based rewards estimate how far an observation has advanced along a nominal successful trajectory, but may remain high after an incorrect transition. We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations. Given the visual history up to a transition start, the model predicts the latent displacement associated with successful execution and scores the observed displacement through signed directional and symmetric magnitude agreement. This transition-level comparison penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress. Dream2Reward requires no failure annotations, progress labels, or synthetic negatives, and produces a dense causal reward. Across mechanism diagnostics and shared-trajectory evaluations, it provides stronger success-failure separation and more informative feedback on low-quality behavior than progress-based alternatives. Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance, including in real-robot manipulation. These results show that comparing realized motion with predicted successful change provides an effective way to convert positive demonstrations into dense rewards for robot learning.

Related papers

• Long-Horizon Language Model Reinforcement Learning via Progressive Point Matching [70.73681123064533]
  We show that a simple and unbiased dense reward formulation, which we term progressive point matching, scales exponentially more efficiently to long-horizon tasks.<n>On extremely hard math reasoning problems, sparse outcome rewards cannot make any progress, whereas segment-level rewards enable improvements at larger test-time token budgets.
  arXiv  Detail & Related papers  (2026-09-07T10:21:41Z)
• Phase-and-First-Arrival VLM Feedback for Sparse-Reward Reinforcement Learning in Surgical Manipulation [18.513281440175113]
  In sparse-reward reinforcement learning, terminal rewards collapse such attempts to the same outcome.<n>We introduce phase-and-first-arrival feedback: one VLM query per recorded episode identifies the furthest visually verified task phase.<n>Across five simulated tasks, our method reaches 75.2% mean success, compared with 52.1% for a reward based on Contrastive Language-Image Pre-training.
  arXiv  Detail & Related papers  (2026-09-07T08:29:59Z)
• Progress Reward Modeling for Robotic Learning: A Comprehensive Survey [18.0494714278744]
  A terminal success signal only tells the robot whether the task is completed.<n>Recent studies have increasingly explored progress rewards that provide feedback during task execution.<n>Existing methods use different observations, goal specifications, output signals, supervision sources, and evaluation protocols.<n>This makes it difficult to compare them and understand what their results actually validate.
  arXiv  Detail & Related papers  (2026-07-22T19:29:52Z)
• DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation [67.08835970838996]
  Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning.<n>Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels.<n>We introduce DenseReward, a dense robotic reward model that addresses both challenges.
  arXiv  Detail & Related papers  (2026-07-14T17:59:29Z)
• Stage-Transition Dense Reward Modeling for Reinforcement Learning [22.634812963829194]
  Reinforcement learning for long-horizon robotic manipulation is often limited by sparse and delayed rewards.<n>This work proposes Stage-Transition Dense Reward (STDR), a visual reward-learning framework that converts unstructured expert videos into logically grounded dense rewards for training RL agents from scratch.<n>STDR delivers two complementary learning signals during online training: (i) stage-transition feedback that provides goal-directed reward, and (ii) within-stage progress feedback that supplies fine-grained guidance toward completing each stage.
  arXiv  Detail & Related papers  (2026-06-30T09:07:19Z)
• Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation [38.88183703128359]
  We propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality robot demonstrations.<n>ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning.<n> Experiments on real-robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines.
  arXiv  Detail & Related papers  (2026-06-23T14:27:10Z)
• Perfect Demo Makes Poor Teacher: Learning Robust Alignment from Critical Motion Segments [53.85970147743299]
  Expert demonstrations are widely assumed to be the gold standard for robot imitation learning.<n>Yet for fine-grained manipulation such as insertion, stacking, and alignment, we uncover a counterintuitive failure mode: fluent demonstrations can be poor teachers.<n>We introduce STAIR, a compact dynamic feature that bridges the vision-language model and the action expert, distilling the short-horizon motion already recorded in each trajectory into dense, motion-aware supervision.
  arXiv  Detail & Related papers  (2026-06-14T04:15:29Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation [57.07494675832939]
  Most existing vision-language-action (VLA) models for robotic manipulation lack progress awareness.<n>In this work, we investigate the estimation and integration of task progress, proposing a novel model named textbf vla<n>Experiments on the CALVIN and LIBERO benchmarks, alongside real-world robot deployment, consistently demonstrate substantial improvements in success rates.
  arXiv  Detail & Related papers  (2026-03-29T12:38:11Z)
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning [50.20267980386502]
  We learn a dense, token-level reward model for process supervision directly from expert demonstrations.<n>The learned reasoning reward serves two complementary roles: (i) it provides step-level feedback to optimise a reasoning policy during training; and (ii) it functions at inference as a critic to rerank sampled traces under fixed compute budgets.
  arXiv  Detail & Related papers  (2025-10-02T09:55:26Z)
• Social NCE: Contrastive Learning of Socially-aware Motion Representations [87.82126838588279]
  Experimental results show that the proposed method dramatically reduces the collision rates of recent trajectory forecasting, behavioral cloning and reinforcement learning algorithms. Our method makes few assumptions about neural architecture designs, and hence can be used as a generic way to promote the robustness of neural motion models.
  arXiv  Detail & Related papers  (2020-12-21T22:25:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.