SelfWAM: A Self-Grounded Unified World Action Model for Fast Robot Control

• URL: http://arxiv.org/abs/2608.00725v1
• Date: Sat, 01 Aug 2026 15:47:29 GMT
• Title: SelfWAM: A Self-Grounded Unified World Action Model for Fast Robot Control
• Abstract summary: World Action Models (WAMs) improve robot policy learning by jointly modeling actions and future observations.<n>We introduce SelfWAM, a unified self-grounded WAM built on a modality-specialized Mixture-of-Transformers (MoT) architecture.<n>We show that SelfWAM produces more action-sensitive futures and preserves fast policy inference, while improving policy performance.
• Score: 38.11007351456953
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: World Action Models (WAMs) improve robot policy learning by jointly modeling actions and future observations. However, conditioning future prediction only on the task prompt and observation context risks capturing generic task progression rather than the action-specific consequences of the executed action. We introduce SelfWAM, a unified self-grounded WAM built on a modality-specialized Mixture-of-Transformers (MoT) architecture that jointly predicts actions, action-conditioned future RGB frames, and robot self-masks, thereby grounding future prediction in the robot's visible body and its action-induced motion. During joint training, SelfWAM allows future visual queries to attend to a clean copy of the demonstrated action, turning the video branch into an action-specific consequence model while leaving the fast action-only inference path unchanged. To focus video learning on action-relevant visual changes, we use prompt-specific objectives for future robot self-mask prediction, which removes appearance details and provides a target whose temporal evolution is tightly coupled with the conditioning action. Together, clean-action conditioning and future self-mask supervision make future predictions more directly reflect how the executed action changes the robot's visible motion and the surrounding scene. Experiments on RoboTwin 2.0 and real-world manipulation tasks show that SelfWAM produces more action-sensitive futures and preserves fast policy inference, while improving policy performance.

Related papers

• Foresight Without Seeing: Latent Futures for World Action Models [31.423405133926334]
  We propose ForeWAM, a dynamics-conditioned direct-policy WAM that provides predictive context for action generation without decoding future videos.<n>We also introduce dynamics supervised by a frozen latent action teacher, encouraging the implicit future states to capture interaction-induced transitions such as object motion, contact changes, and task progress.<n>Without embodied robot data pretraining, the standard and accelerated variants of ForeWAM achieve average success rates of 96.7% and 96.9% on LIBERO, respectively.
  arXiv  Detail & Related papers  (2026-08-12T03:27:43Z)
• Masked Visual Actions for Unified World Modeling [96.12988421978463]
  We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video.<n>Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome.
  arXiv  Detail & Related papers  (2026-07-21T17:59:11Z)
• WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos [38.959415159248046]
  WALA is a framework for learning executable latent actions from both action-labeled demonstrations and action-free videos.<n>We show that WALA achieves strong performance on RoboTwin, sets a new state-of-the-art result on RoboCasa with 75.2% average success.
  arXiv  Detail & Related papers  (2026-07-13T11:02:04Z)
• LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies [31.88655699486955]
  We present LaWAM, a latent-action-conditioned Latent World Model (LaWM)<n>LaWAM exposes predictive dynamics to robot policies through compact latent visual subgoals instead of reconstructed future video.<n>LaWAM runs in 187 ms per action-chunk prediction and achieves up to 24x lower wall-clock latency than pixel-space WAMs.
  arXiv  Detail & Related papers  (2026-06-14T12:06:58Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• GigaWorld-Policy: An Efficient Action-Centered World--Action Model [50.107640832046464]
  GigaWorld-Policy is an action-centered WAM that learns 2D pixel-action dynamics while enabling efficient action decoding, with optional video generation.<n> Experimental results on real-world robotic platforms show that GigaWorld-Policy runs 9x faster than the leading WAM baseline, Motus.
  arXiv  Detail & Related papers  (2026-03-18T00:52:02Z)
• Physical Autoregressive Model for Robotic Manipulation without Action Pretraining [65.8971623698511]
  We build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR)<n>PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining.<n>Experiments on the ManiSkill benchmark show that PAR achieves a 100% success rate on the PushCube task.
  arXiv  Detail & Related papers  (2025-08-13T13:54:51Z)
• Reimagination with Test-time Observation Interventions: Distractor-Robust World Model Predictions for Visual Model Predictive Control [51.14656121641822]
  World models enable robots to "imagine" future observations given current observations and planned actions.<n>Novel visual distractors can corrupt action outcome predictions, causing downstream failures when robots rely on the world model imaginations for planning or action verification.<n>We propose Reimagination with Observation Intervention (ReOI), a simple yet effective test-time strategy that enables world models to predict more reliable action outcomes.
  arXiv  Detail & Related papers  (2025-06-19T19:41:29Z)
• Future Frame Prediction for Robot-assisted Surgery [57.18185972461453]
  We propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools.
  arXiv  Detail & Related papers  (2021-03-18T15:12:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.