Agentic World Action Models
Abstract Overview
This paper introduces AW-0, an agentic world action model that combines code-based planning with future video prediction and low-level action generation in a single mixture-of-transformers architecture. The model treats code, future visual states, and robot actions as three coupled generative streams that are jointly denoised, so structured skill plans can guide both prediction and control. The authors also describe an automated annotation pipeline that labels robot and human manipulation trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. AW-0 is pretrained on roughly 2,000 hours of robot data and 1,000 hours of egocentric human manipulation data, then evaluated in simulation and on real bimanual manipulation tasks.
Novelty
The main novelty is the integration of a visually grounded code expert directly inside a world action model, rather than using code only as an external planner. The work also contributes an automatic program-annotation and skill-discovery pipeline, plus a hybrid execution scheme that can switch between direct action chunks and executable skill programs at deployment.
Results
On RoboTwin 2.0, AW-0 achieves the best reported average success rate across clean and randomized settings at 50.7%, including 80.8% in the clean setting; its hybrid mode improves robustness under randomization to 29.2%. On 8 real-world tasks across parallel-gripper and dexterous-hand platforms, AW-0 reaches a 71.1% average rubric score, outperforming the strongest baseline by 25.1 points. Ablations further show that code-first joint denoising and ViT-based visual conditioning improve task performance, and joint denoising is both more accurate than autoregressive code generation and faster at inference.
Key Points
- AW-0 jointly models code, future video, and robot actions, allowing reusable skill programs to shape both world prediction and control generation.
- The paper introduces an automated curation pipeline that annotates demonstrations with program slices and distills recurring behaviors into reusable skill libraries.
- Empirically, AW-0 improves manipulation performance in both simulation and real-world bimanual tasks, while hybrid code execution particularly strengthens robustness under distribution shift.
References
- arXiv: https://arxiv.org/abs/2610.30136v1
- Fugu-MT: https://fugumt.com/fugumt/paper_check/2610.30136v1
- Hugging Face Papers: https://huggingface.co/papers/2610.30136
- Project: https://agentic-wam.github.io/