ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts

• URL: http://arxiv.org/abs/2607.28993v1
• Date: Fri, 31 Jul 2026 03:44:56 GMT
• Title: ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts
• Abstract summary: Training-Distribution Hallucination is a recurring phenomenon in which futures conditioned on visually shifted observations hallucinate training-domain content rather than remain faithful to the current scene.<n>We propose Semantic-Temporal WAM (ST-WAM) to improve action robustness by using DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining fine-grained VAE dynamics.
• Score: 24.959404195743744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World Action Models (WAMs) have emerged as a promising paradigm by jointly modeling robot actions and future visual dynamics. However, their reliance on pixel-generative future supervision can entangle action-relevant state transitions with task-irrelevant visual content, limiting robustness under visual distribution shifts. We identify Training-Distribution Hallucination, a recurring phenomenon in which futures conditioned on visually shifted observations hallucinate training-domain content rather than remain faithful to the current scene. A controlled frame-triplet diagnosis further shows that DINOv3 features remain more stable across visual shifts while better preserving task-state distinctions than Wan-VAE latents. Rather than correcting the predicted futures, we propose Semantic-Temporal WAM (ST-WAM) to improve action robustness by using DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining fine-grained VAE dynamics. Its Dual-Space Future Experts (DSFE) jointly predict future VAE latents and DINO features, while Current-Anchored Intent Retrieval (CAIR) retrieves task-relevant evidence from recent DINO history under the current visual-language context. ST-WAM is trained end-to-end without additional embodied pretraining or task-specific annotations, and requires no explicit future generation at inference. It achieves 98.7% on LIBERO and 92.8% on RoboTwin 2.0; more importantly, compared with Fast-WAM, it improves zero-shot LIBERO-Plus performance by 21.3 percentage points and more than doubles real-world success under visual shifts from 25.8% to 61.5%. These results demonstrate that semantic-temporal modeling effectively complements pixel-generative dynamics for robust manipulation.

Related papers

• Learning to Use Imagination: Progress-Conditioned Future Utilization for World Action Models [72.02240114225107]
  World Action Models (WAMs) extend Vision-Language-Action (VLA) models by incorporating future visual dynamics into action generation.<n>Existing WAMs often utilize imagined futures with limited adaptation to evolving execution progress, potentially introducing distracting or unreliable predictive cues.<n>We propose ProWAM, a Progress-Conditioned World Action Model that introduces execution progress as an explicit intermediate representation for imagination utilization.
  arXiv  Detail & Related papers  (2026-09-06T12:46:20Z)
• Foresight Without Seeing: Latent Futures for World Action Models [31.423405133926334]
  We propose ForeWAM, a dynamics-conditioned direct-policy WAM that provides predictive context for action generation without decoding future videos.<n>We also introduce dynamics supervised by a frozen latent action teacher, encouraging the implicit future states to capture interaction-induced transitions such as object motion, contact changes, and task progress.<n>Without embodied robot data pretraining, the standard and accelerated variants of ForeWAM achieve average success rates of 96.7% and 96.9% on LIBERO, respectively.
  arXiv  Detail & Related papers  (2026-08-12T03:27:43Z)
• Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Models [40.0682886050991]
  World Action Models (WAMs) improve robot manipulation by learning how the environment evolves beyond the current observation.<n>Joint-WAMs preserve future-aware representations during inference but incur prohibitive computation costs.<n>We show that inference-time future conditioning is critical for generalization under distribution shifts.
  arXiv  Detail & Related papers  (2026-08-05T03:18:58Z)
• FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptation [14.178611843420683]
  We introduce FOCA, a future-oriented conditioning framework for data-efficient VLA adaptation.<n> FOCA combines explicit prediction of task-grounded future interaction embeddings with implicit alignment to future goal observations.<n>Experiments demonstrate FOCA achieves 95.7% success with 20 demonstrations on LIBERO, improves 7-12% on RoboCasa, and delivers up to 26% absolute gains on real robots.
  arXiv  Detail & Related papers  (2026-06-18T18:54:51Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• When to Trust Imagination: Adaptive Action Execution for World Action Models [42.51856318901667]
  World Action Models (WAMs) have recently emerged as a promising paradigm for robotic manipulation by jointly predicting future visual observations and future actions.<n>Current WAMs execute a fixed number of predicted actions after each model inference, leaving the robot blind to whether the imagined future remains consistent with the actual physical rollout.<n>We formulate adaptive WAM execution as a future-reality verification problem: the robot should execute longer when the WAM-predicted future remains reliable, and replan earlier when reality deviates from imagination.
  arXiv  Detail & Related papers  (2026-05-07T13:18:28Z)
• LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning [51.969318585152116]
  LatentPilot exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics.<n>We propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution.<n> Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene.
  arXiv  Detail & Related papers  (2026-03-31T02:21:59Z)
• FUTURE-VLA: Forecasting Unified Trajectories Under Real-time Execution [27.399648455932397]
  FUTURE-VLA is a unified architecture that reformulates long-horizon control and future forecasting as a monolithic sequence-generation task.<n> FUTURE-VLA achieves success rates of 99.2% on LIBERO, 75.4% on RoboTwin, and 78.0% on a real-world Piper platform.
  arXiv  Detail & Related papers  (2026-02-05T14:27:43Z)
• From Observations to States: Latent Time Series Forecasting [65.98504021691666]
  We propose Latent Time Series Forecasting (LatentTSF), a novel paradigm that shifts TSF from observation regression to latent state prediction.<n>Specifically, LatentTSF employs an AutoEncoder to project observations at each time step into a higher-dimensional latent state space.<n>Our proposed latent objectives implicitly maximize mutual information between predicted latent states and ground-truth states and observations.
  arXiv  Detail & Related papers  (2026-01-30T20:39:44Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• LaST$_{0}$: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Model [102.60980325911106]
  We propose a framework that enables efficient reasoning before acting through a Latent Spatio-Temporal Chain-of-Thought (CoT)<n>Specifically, we introduce a token-efficient latent CoT space that models future visual dynamics, 3D structural information, and robot proprioceptive states, and further extends these representations across time to enable temporally consistent implicit reasoning trajectories.
  arXiv  Detail & Related papers  (2026-01-08T18:59:53Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.