TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance

• URL: http://arxiv.org/abs/2509.26627v1
• Date: Tue, 30 Sep 2025 17:58:20 GMT
• Title: TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance
• Abstract summary: TimeRewarder is a simple yet effective reward learning method that derives progress estimation signals from passive videos.<n>We show that TimeRewarder dramatically improves RL for sparse-reward tasks, achieving nearly perfect success in 9/10 tasks with only 200,000 interactions per task with the environment.
• Score: 36.22149703563646
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Designing dense rewards is crucial for reinforcement learning (RL), yet in robotics it often demands extensive manual effort and lacks scalability. One promising solution is to view task progress as a dense reward signal, as it quantifies the degree to which actions advance the system toward task completion over time. We present TimeRewarder, a simple yet effective reward learning method that derives progress estimation signals from passive videos, including robot demonstrations and human videos, by modeling temporal distances between frame pairs. We then demonstrate how TimeRewarder can supply step-wise proxy rewards to guide reinforcement learning. In our comprehensive experiments on ten challenging Meta-World tasks, we show that TimeRewarder dramatically improves RL for sparse-reward tasks, achieving nearly perfect success in 9/10 tasks with only 200,000 interactions per task with the environment. This approach outperformed previous methods and even the manually designed environment dense reward on both the final success rate and sample efficiency. Moreover, we show that TimeRewarder pretraining can exploit real-world human videos, highlighting its potential as a scalable approach path to rich reward signals from diverse video sources.

Related papers

• GORDON: Graph-based Object-centric Rewards for Decomposition of Long-Horizon Manipulation [15.203442273197455]
  GORDON is a graph-based object-centric reward learning framework.<n>It learns dense rewards from action-free video demonstrations.<n>It achieves an average success rate of 74.4% across the long-horizon tasks.
  arXiv  Detail & Related papers  (2026-08-04T14:43:41Z)
• DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation [67.08835970838996]
  Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning.<n>Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels.<n>We introduce DenseReward, a dense robotic reward model that addresses both challenges.
  arXiv  Detail & Related papers  (2026-07-14T17:59:29Z)
• Learning Process Rewards via Success Visitation Matching for Efficient RL [54.67547527674048]
  We propose a simple approach to transform a sparse outcome reward into a dense process reward.<n>Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes.<n>By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion.
  arXiv  Detail & Related papers  (2026-06-22T17:30:24Z)
• RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation [12.756156905476503]
  Reinforcement learning for robot manipulation is often bottlenecked by reward design.<n>We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward.<n>RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding.
  arXiv  Detail & Related papers  (2026-06-20T13:03:21Z)
• On-Robot Reinforcement Learning with Goal-Contrastive Rewards [24.415607337006968]
  Reinforcement Learning (RL) has the potential to enable robots to learn from their own actions in the real world.<n>We propose GCR (Goal-Contrastive Rewards), a dense reward function learning method that can be trained on passive video demonstrations.<n>GCR combines two loss functions, an implicit value loss function that models how the reward increases when traversing a successful trajectory, and a goal-contrastive loss that discriminates between successful and failed trajectories.
  arXiv  Detail & Related papers  (2024-10-25T22:11:54Z)
• Affordance-Guided Reinforcement Learning via Visual Prompting [51.361977466993345]
  Keypoint-based Affordance Guidance for Improvements (KAGI) is a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL.<n>On diverse real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 30K online fine-tuning steps.
  arXiv  Detail & Related papers  (2024-07-14T21:41:29Z)
• DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks [26.730889757506915]
  We propose DrS (Dense reward learning from Stages), a novel approach for learning reusable dense rewards for multi-stage tasks. By leveraging the stage structures of the task, DrS learns a high-quality dense reward from sparse rewards and demonstrations if given. Experiments on three physical robot manipulation task families with 1000+ task variants demonstrate that our learned rewards can be reused in unseen tasks.
  arXiv  Detail & Related papers  (2024-04-25T17:28:33Z)
• DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing [60.21269454707625]
  DreamSmooth learns to predict a temporally-smoothed reward, instead of the exact reward at the given timestep. We show that DreamSmooth achieves state-of-the-art performance on long-horizon sparse-reward tasks.
  arXiv  Detail & Related papers  (2023-11-02T17:57:38Z)
• Go Beyond Imagination: Maximizing Episodic Reachability with World Models [68.91647544080097]
  In this paper, we introduce a new intrinsic reward design called GoBI - Go Beyond Imagination. We apply learned world models to generate predicted future states with random actions. Our method greatly outperforms previous state-of-the-art methods on 12 of the most challenging Minigrid navigation tasks.
  arXiv  Detail & Related papers  (2023-08-25T20:30:20Z)
• Reward Relabelling for combined Reinforcement and Imitation Learning on sparse-reward tasks [2.0305676256390934]
  We present a new method to leverage demonstrations and episodes collected online in any sparse-reward environment with any off-policy algorithm. Our method is based on a reward bonus given to demonstrations and successful episodes, encouraging expert imitation and self-imitation. Our experiments focus on manipulation robotics, specifically on three tasks for a 6 degrees-of-freedom robotic arm in simulation.
  arXiv  Detail & Related papers  (2022-01-11T08:35:18Z)
• Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation [55.31301153979621]
  We tackle real-world long-horizon robot manipulation tasks through skill discovery. We present a bottom-up approach to learning a library of reusable skills from unsegmented demonstrations. Our method has shown superior performance over state-of-the-art imitation learning methods in multi-stage manipulation tasks.
  arXiv  Detail & Related papers  (2021-09-28T16:18:54Z)
• A Study on Dense and Sparse (Visual) Rewards in Robot Policy Learning [19.67628391301068]
  We study the performance of multiple state-of-the-art deep reinforcement learning algorithms under different types of reward. Our results show that visual dense rewards are more successful than visual sparse rewards and that there is no single best algorithm for all tasks.
  arXiv  Detail & Related papers  (2021-08-06T17:47:48Z)
• Semi-supervised reward learning for offline reinforcement learning [71.6909757718301]
  Training agents usually requires reward functions, but rewards are seldom available in practice and their engineering is challenging and laborious. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards.
  arXiv  Detail & Related papers  (2020-12-12T20:06:15Z)
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards [1.2691047660244335]
  We propose a practical self-imitation learning method named Self-Imitation Learning with Constant Reward (SILCR) Our method assigns the immediate rewards at each timestep with constant values according to their final episodic rewards. We demonstrate the effectiveness of our method in some challenging continuous robotics control tasks in MuJoCo simulation.
  arXiv  Detail & Related papers  (2020-10-14T11:12:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.