Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
Abstract Overview
This paper frames zero-shot cross-task robotic manipulation as an in-context learning problem, where an unseen task is specified at deployment time by a human demonstration video rather than only by language. The authors propose Zero-WAM, a causal video-action model that predicts future robot video chunks and aligned actions while conditioning on either language or human-video prompts. To make such training scalable, they introduce HumanGen, an automatically generated dataset of 74.2K human-robot in-context pairs spanning 8.6K tasks, together with task-balanced robotic pretraining data sampled at the task level. They also add an in-context future chunk prediction objective to reduce shortcut learning from seen-task robot histories and encourage use of the human-video prompt.
Novelty
The distinctive contribution is the use of human videos as in-context task specifications for zero-shot robotic task generalization within a single causal world-action model. The work also introduces an automatic pipeline for generating large-scale semantically matched human-robot pairs and an auxiliary future-chunk prediction objective designed specifically to make the model rely more on the in-context video.
Results
In RoboTwin 2.0 simulation, Zero-WAM achieves a 46.95% average success rate across seven unseen tasks, improving over LingBot-VA by 29.50 percentage points and over a Wan-based baseline by 35.97 points. In real-world evaluations on unseen task configurations, it reaches 53.3% success for object-to-container placement, 33.3% for three-object sequential manipulation, and 16.7% for two-table-leg insertion, outperforming the language-conditioned LingBot-VA baseline in all three settings.
Key Points
- Zero-WAM conditions robot execution on human demonstration videos as in-context task prompts, aiming to generalize to tasks not seen during robot training.
- The paper introduces HumanGen, a 74.2K-pair dataset across 8.6K tasks, built by automatically converting task-sampled robot trajectories into semantically matched human videos.
- An in-context future chunk prediction objective and task-balanced robotic pretraining are shown to improve unseen-task performance, including a 46.95% average success rate on seven unseen RoboTwin tasks.