DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation

• URL: http://arxiv.org/abs/2607.13033v1
• Date: Tue, 14 Jul 2026 17:59:29 GMT
• Title: DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation
• Abstract summary: Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning.<n>Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels.<n>We introduce DenseReward, a dense robotic reward model that addresses both challenges.
• Score: 67.08835970838996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning. However, its practical adoption remains bottlenecked by the lack of reliable vision-language reward models that provide dense and informative feedback. Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels. Collecting failure trajectories typically requires laborious human effort, while pseudo-failures constructed by relabeling successful demonstrations fail to capture the diverse physical failure modes that arise during robot execution. Meanwhile, existing reward models often predict sparse binary or trajectory-level rewards, which provide limited guidance for efficient policy optimization. We introduce DenseReward, a dense robotic reward model that addresses both challenges. To train DenseReward, we develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. DenseReward predicts dense frame-level reward scores from visual observations and language instructions, enabling fine-grained estimation of task progress throughout an episode. Experiments show that DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation. We further demonstrate that DenseReward provides effective reward guidance for downstream model predictive control and reinforcement learning. We release the dataset, trained reward models, and evaluation suite to support the development of failure-aware dense reward modeling for robot learning.

Related papers

• Dream2Reward: Transition-Alignment Reward Models from Positive Demonstrations for Robotic Manipulation [51.43645327777772]
  We introduce Dream2Reward, which learns a language-conditioned successful latent transition field from positive demonstrations.<n>It penalizes wrong-direction, overshooting, and stagnant motion even when the resulting observation appears to show progress.<n>Across online and offline policy learning, the same frozen reward model reduces reward hacking and supports stronger downstream performance.
  arXiv  Detail & Related papers  (2026-08-19T10:44:20Z)
• Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation [38.88183703128359]
  We propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality robot demonstrations.<n>ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning.<n> Experiments on real-robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines.
  arXiv  Detail & Related papers  (2026-06-23T14:27:10Z)
• RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation [12.756156905476503]
  Reinforcement learning for robot manipulation is often bottlenecked by reward design.<n>We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward.<n>RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding.
  arXiv  Detail & Related papers  (2026-06-20T13:03:21Z)
• From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models [40.30751311757326]
  Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics.<n>Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models.<n>We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones.
  arXiv  Detail & Related papers  (2026-05-22T16:04:22Z)
• Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons [69.87766750714945]
  General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations.<n>We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision.<n>Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints.
  arXiv  Detail & Related papers  (2026-03-02T17:38:58Z)
• Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [53.20969621498248]
  We propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures.<n>We construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail.<n>We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection.
  arXiv  Detail & Related papers  (2025-12-01T17:57:27Z)
• GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning [79.68241687396603]
  We propose GenFlowRL, which derives shaped rewards from generated flow trained from diverse cross-embodiment datasets.<n>Experiments on 10 manipulation tasks, both in simulation and real-world cross-embodiment evaluations, demonstrate that GenFlowRL effectively leverages manipulation features extracted from generated object-centric flow.
  arXiv  Detail & Related papers  (2025-08-14T20:19:20Z)
• Affordance-Guided Reinforcement Learning via Visual Prompting [51.361977466993345]
  Keypoint-based Affordance Guidance for Improvements (KAGI) is a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL.<n>On diverse real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 30K online fine-tuning steps.
  arXiv  Detail & Related papers  (2024-07-14T21:41:29Z)
• DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing [60.21269454707625]
  DreamSmooth learns to predict a temporally-smoothed reward, instead of the exact reward at the given timestep. We show that DreamSmooth achieves state-of-the-art performance on long-horizon sparse-reward tasks.
  arXiv  Detail & Related papers  (2023-11-02T17:57:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.