Fugu-MT: arxivの論文翻訳
Daily Report / 2026-07-21

Daily Paper Briefing

2026-07-21 1 Paper

Masked Visual Actions for Unified World Modeling

Authors Hadi Alzayer, Wenlong Huang, Haonan Chen, Christopher Luey, Lvmin Zhang, Maneesh Agrawala, Gordon Wetzstein, Li Fei-Fei, Yilun Du, Jiajun Wu, Jia-Bin Huang
Affiliations Stanford University / University of Maryland / Harvard University
Categories Method / World Modeling / Unified visual and action modeling, Application / Robot Manipulation / Low-level motion prediction and control, Evaluation / Motion Prediction / Scene response prediction evaluation
License CC BY 4.0

Abstract Overview

This paper introduces Masked Visual Actions, a pixel-space control interface for video world models in which action is represented as a partially revealed trajectory of an entity in the video. The same finetuned video model can be used in two modes: revealing robot motion makes it predict scene response as a forward model, while revealing desired object motion makes it infer robot behavior as an inverse model. The method is implemented by finetuning a pretrained video model with masked examples from real and simulated robot data, using lightweight adaptation rather than full retraining. Experiments evaluate controllable video generation, policy evaluation, model-based planning, and action extraction in both simulation and real-world settings.

Novelty

The main novelty is expressing robot action directly in pixel space as masked trajectories, aligning the control signal with the representations learned by pretrained video models. This masking interface also lets a single checkpoint serve as a forward model, inverse model, or unconditional generator by changing which entities are revealed, including zero-shot inverse use despite training only on forward-style masks.

Results

With 15 hours of finetuning data, the model reports better video-generation metrics than image-to-video, track-conditioned, and raw-action baselines on both seen and unseen embodiments, including strong generalization on the BEHAVIOR dataset. In downstream use, imagined policy rollouts correlate strongly with ground-truth task outcomes in RoboCasa (reported r = 0.982), planning improves task success by ranking candidate futures, and the inverse pipeline reaches a 90% success rate on the CoffeeServeMug task.

Key Points

  1. Masked Visual Actions conditions a video model using dense, spatially aligned masked trajectories rather than embodiment-specific action vectors or sparse visual cues.
  2. A single finetuned checkpoint supports both forward prediction of scene effects from robot motion and inverse synthesis of robot behavior from desired object motion.
  3. The method shows practical utility beyond video generation, including policy evaluation, model-based planning, and action extraction across simulation and real-world robot settings.

References

Project
This page was created using generative AI such as GPT-5, Claude Opus 4, Gemini 3, Gemini 3.1 Flash Image, and their higher-end successor versions. No guarantee can be made regarding its contents.