$τ_0$-WM: A Unified Video-Action World Model for Robotic Manipulation

• URL: http://arxiv.org/abs/2606.01027v1
• Date: Sun, 31 May 2026 05:35:36 GMT
• Title: $τ_0$-WM: A Unified Video-Action World Model for Robotic Manipulation
• Abstract summary: We present a unified video-action world model that integrates policy learning, video prediction, and action evaluation.<n>The model is trained on approximately $27,300$ hours of real-robot teleoperation.
• Score: 45.040666672458634
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Robotic manipulation requires models that generate executable actions while anticipating and evaluating their future consequences before physical execution. We present $τ_0$-World Model ($τ_0$-WM), a unified video-action world model that integrates policy learning, video prediction, and action evaluation within a single future-predictive framework. Built on a shared video diffusion backbone, $τ_0$-WM provides two complementary interfaces. First, a video action model jointly predicts future visual latents and continuous action chunks from multi-view observations, language instructions, and robot state. Second, an action-conditioned video simulator rolls out candidate action chunks into multi-view futures and predicts dense task-progress scores. The model is trained on approximately $27{,}300$ hours of real-robot teleoperation, UMI-style interaction, egocentric human videos, and rollout or failure trajectories using modality-specific supervision masks. At inference time, $τ_0$-WM uses test-time computation to sample action candidates, rank them with re-denoising consistency, and invoke simulator-based rectification for low-quality candidates. On challenging long-horizon and fine-grained robotic manipulation tasks, $τ_0$-WM shows superior performance over other relevant baselines.

Related papers

• TrAct: Bridging Robot Control and Visual Prediction with Visual Tracks [22.702826122425268]
  We propose TrAct, a world-model-based robot decision-making framework that uses visual tracks as an intermediate interface between control and prediction.<n>TrAct consists of three components: a Vision-Language-Action-and-Track model (VLAT) that jointly predicts candidate actions and corresponding visual tracks from the current observation and language instruction; a track-conditioned world model (TWM) that predicts future visual outcomes conditioned on the proposed tracks; and a vision-language reward model (VLAC) that scores the predicted outcomes.
  arXiv  Detail & Related papers  (2026-08-25T06:00:01Z)
• Masked Visual Actions for Unified World Modeling [96.12988421978463]
  We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video.<n>Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome.
  arXiv  Detail & Related papers  (2026-07-21T17:59:11Z)
• World Action Models: A Survey [100.95337034529263]
  World Action Models (WAMs) are embodied predictive-action models that make a forecast of the future available to action.<n>Recent WAMs repurpose large video generation models, and a parallel line relies on language or vision-language backbones without a video-generation core.<n>This survey gives the field a common account. It first clarifies these boundaries, then organizes existing works through two complementary views.
  arXiv  Detail & Related papers  (2026-06-18T17:05:19Z)
• From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data [71.22409934108924]
  Human videos are abundant and capture rich interactions, providing diverse semantic and physical cues for real-world manipulation.<n>This survey provides a unified view of how human videos are transformed into effective knowledge for Vision-Language-Action (VLA) models.<n>We highlight three key open challenges in this area: structuring unstructured videos into training-ready episodes, grounding video-derived supervision into robot-executable actions under embodiment and viewpoint heterogeneity, and designing evaluation protocols that better predict real-world deployment performance and transfer efficiency.
  arXiv  Detail & Related papers  (2026-05-18T06:19:16Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• GigaWorld-Policy: An Efficient Action-Centered World--Action Model [50.107640832046464]
  GigaWorld-Policy is an action-centered WAM that learns 2D pixel-action dynamics while enabling efficient action decoding, with optional video generation.<n> Experimental results on real-world robotic platforms show that GigaWorld-Policy runs 9x faster than the leading WAM baseline, Motus.
  arXiv  Detail & Related papers  (2026-03-18T00:52:02Z)
• Large Video Planner Enables Generalizable Robot Control [117.49024534548319]
  General-purpose robots require decision-making models that generalize across diverse tasks and environments.<n>Recent works build robot foundation models by extending multimodal large language models (LMs) with action outputs, creating vision--action (VLA) systems.<n>We explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models.
  arXiv  Detail & Related papers  (2025-12-17T18:35:54Z)
• FLARE: Robot Learning with Implicit World Modeling [87.81846091038676]
  $textbfFLARE$ integrates predictive latent world modeling into robot policy learning.<n>$textbfFLARE$ achieves state-of-the-art performance, outperforming prior policy learning baselines by up to 26%.<n>Our results establish $textbfFLARE$ as a general and scalable approach for combining implicit world modeling with high-frequency robotic control.
  arXiv  Detail & Related papers  (2025-05-21T15:33:27Z)
• Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression [23.99292102237088]
  We propose Heterogeneous Masked Autoregression (HMA) for modeling action-video dynamics.<n>After post-training, this model can be used as a video simulator for evaluating policies and generating synthetic data.
  arXiv  Detail & Related papers  (2025-02-06T18:38:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.