RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

• URL: http://arxiv.org/abs/2608.09853v1
• Date: Mon, 10 Aug 2026 17:09:37 GMT
• Title: RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance
• Abstract summary: General-purpose reward models are increasingly the bottleneck for scaling robot learning.<n>We introduce RynnValue, an open-source value foundation model for robotic manipulation.<n>RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations.
• Score: 84.26294415726723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce RynnValue, an open-source value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, RynnValue attains an average Kendall's tau_a of 0.675 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.

Related papers

• ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training [66.72708893922605]
  Post-training of large multimodal models should add new capabilities while preserving those from pre-training.<n>SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting.<n>We introduce ReDraft, which obtains both from the model's own failures.
  arXiv  Detail & Related papers  (2026-09-15T04:59:03Z)
• LM-X: Explainable Action Modeling with Progress, Event, and Uncertainty Prediction for Generalist Robot Manipulation [28.431160101245837]
  Generalist vision--language--action (VLA) policies learn long-horizon behavior mainly through short-horizon action prediction.<n>We introduce LM-X, which organizes prediction across task, event, and motor scales without claiming anatomical correspondence.
  arXiv  Detail & Related papers  (2026-08-26T13:05:29Z)
• On-Policy Self-Distillation in Diffusion Models [89.32656931795293]
  Reinforcement learning can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change.<n>We introduce DiffusionOPSD as an on-policy self-distillation framework that converts image-level reward guidance into explicit targets for clean-output predictions at sampled queries.<n>Across SD 3.5-M and the step-distilled Z-Image-Turbo, our approach achieves the best final held-out scores in 19 of 20 reward-matched settings across two backbones and ten evaluators.
  arXiv  Detail & Related papers  (2026-08-25T14:55:24Z)
• ForeTime-VLA: Causal Future-Token Distillation from a World Action Model for Conveyor-Belt Manipulation [6.874305530359895]
  We introduce ForeTime-VLA, a dense pi0.5 policy that distills a future-aware, action-equivalent representation from a frozen Fast-WAM-derived teacher.<n>Online, an eight-frame history encoder predicts this target together with manipulation phase and normalized time-to-transition.<n>Test MAE decreases from 0.134119 to 0.130593 (2.63%); paired-bootstrap 95% CI: 0.82-4.48% improvement, and test L2 decreases by 3.02%, at a 2.46-2.93% latency cost.
  arXiv  Detail & Related papers  (2026-08-21T04:40:41Z)
• Hermite Curves as Trajectory Priors for Vision-Language-Action Models [41.51953284790607]
  We introduce Hermite trajectory priors, parameterizing the chunk trajectory as a piecewise cubic Hermite curve defined by endpoint positions and velocities.<n>We instantiate this fixed operator across discrete autoregressive and continuous generative paradigms.<n>Across simulation benchmarks and real-robot platforms, Hermite Regularization achieves superior performance.
  arXiv  Detail & Related papers  (2026-08-02T14:18:17Z)
• STEAM: Self-Supervised Temporal Ensemble Advantage Modeling for Real-World Robot Learning [76.76959348575764]
  Policy learning requires frame-level advantages that distinguish reliable local progress from failures and regressions.<n>We propose Self-supervised Temporal Temporal Ensemble Advantage Modeling (STEAM), a label-free method that learns such advantages from expert demonstrations.
  arXiv  Detail & Related papers  (2026-06-29T06:19:35Z)
• ESPO: Early-Stopping Proximal Policy Optimization [78.79610718910628]
  ESPO (Early-Stopping Proximal Policy Optimization) detects trajectory failure on-the-fly and terminates rollouts early.<n>On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME2024 (46.28% vs. 45.25%), AMC2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%)
  arXiv  Detail & Related papers  (2026-05-28T12:40:22Z)
• Forecasting Downstream Performance of LLMs With Proxy Metrics [37.61408486519996]
  We propose to construct proxy metrics by aggregating token-level statistics, such as entropy, top-k accuracy, and expert token rank.<n>Across three settings, our proxies consistently outperform loss- and compute-based baselines.
  arXiv  Detail & Related papers  (2026-05-18T16:17:15Z)
• How Far Can Unsupervised RLVR Scale LLM Training? [57.44753418846446]
  Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck.<n>Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear.<n>We classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework.
  arXiv  Detail & Related papers  (2026-03-09T17:38:11Z)
• A Rubric-Supervised Critic from Sparse Real-World Outcomes [87.11204512676193]
  Real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse.<n>We propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling.
  arXiv  Detail & Related papers  (2026-03-04T07:23:54Z)
• Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons [69.87766750714945]
  General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations.<n>We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision.<n>Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints.
  arXiv  Detail & Related papers  (2026-03-02T17:38:58Z)
• Interpretable Machine Learning Approaches to Prediction of Chronic Homelessness [2.294014185517203]
  We introduce a machine learning approach to predict chronic homelessness from de-identified client shelter records. Our model, HIFIS-RNN-MLP, incorporates both static and dynamic features of a client's history to forecast chronic homelessness 6 months into the client's future.
  arXiv  Detail & Related papers  (2020-09-12T15:02:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.