MaskWAM: Unifying Mask Prompting and Prediction for World-Action Models

• URL: http://arxiv.org/abs/2606.13515v1
• Date: Thu, 11 Jun 2026 16:02:42 GMT
• Title: MaskWAM: Unifying Mask Prompting and Prediction for World-Action Models
• Abstract summary: World Action Models (WAMs) present a promising paradigm for robotic control via video prediction.<n>Current WAMs suffer from fundamental spatial bottlenecks.<n>We introduce MaskWAM, an object-centric world-action model.
• Score: 41.48585089405156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World Action Models (WAMs) present a promising paradigm for robotic control via video prediction. However, current WAMs suffer from fundamental spatial bottlenecks: standard text inputs introduce referential ambiguity in cluttered scenes, while unstructured RGB predictions lack semantic grounding and remain biased by task-irrelevant backgrounds. To overcome these limitations, we introduce MaskWAM, an object-centric world-action model. By jointly integrating masks as both explicit inputs and predictions via a unified Mixture of Transformers (MoT), MaskWAM unlocks robust policy generalization. This design provides two key benefits: (1) predicting future masks yields object-centric semantic supervision that suppresses visual noise, significantly enhancing even standard text-conditioned WAMs; and (2) coupling this predictive supervision with first-frame visual prompts, such as target object masks, establishes a precise spatial anchor that substantially reduces language ambiguity. Crucially, as WAMs are inherently vision-driven architectures, direct mask conditioning yields substantially stronger guidance than text alone, establishing a precise and robust paradigm for manipulating unseen objects. Evaluations on LIBERO, RoboTwin, and real-world tasks demonstrate that MaskWAM significantly outperforms baselines in both language-clear and language-ambiguous tasks.

Related papers

• GenEraser: Generalizable Video Object Removal via Balanced Text-Mask Guidance and Decoupled Locator-Preserver [107.6554560318856]
  GenEraser is a novel framework for generalized and high-fidelity video object and effect removal.<n>We introduce a Multi-Conditional Mixture-of-Experts (MC-MoE) paired with Bipartite Text guidance to fully exploit the multimodal priors of Diffusion Transformers.<n>We also propose a Learnable Deep C'FG' Fusion mechanism to adaptively balance the relative dominance of mask and textual conditions.
  arXiv  Detail & Related papers  (2026-05-28T14:58:36Z)
• See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding [41.70825455674463]
  SWIM (See What I Mean) is a training strategy that aligns vision and language representations to enable fine-grained object understanding solely from textual prompts.<n>We show that SWIM substantially improves text-visual alignment and achieves superior performance over visual-prompt-based methods on fine-grained object understanding benchmarks.
  arXiv  Detail & Related papers  (2026-05-18T08:09:37Z)
• Perception-Aware Multimodal Spatial Reasoning from Monocular Images [57.42071289037214]
  spatial reasoning from monocular images is essential for autonomous driving.<n>Current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception.<n>We propose a perception-aware multimodal reasoning framework that equipsVLMs with explicit object-centric grounding ability.
  arXiv  Detail & Related papers  (2026-03-07T02:05:12Z)
• Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models [19.847438086389616]
  Masked Diffusion Language Models have emerged as a promising alternative to Autoregressive Language Models.<n>We show that MDLMs exhibit a strong locality bias, favouring local over distant context.<n>We introduce a mask-agnostic loss function that encourages predictions to remain invariant to the number of appended masks.
  arXiv  Detail & Related papers  (2025-11-26T12:44:29Z)
• Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrieval [15.126709823382539]
  This work advances Contrastive Language-Image Pre-training (CLIP) for person representation learning.<n>We develop a noise-resistant data construction pipeline that leverages the in-context learning capabilities of MLLMs.<n>We introduce the GA-DMS framework, which improves cross-modal alignment by adaptively masking noisy textual tokens.
  arXiv  Detail & Related papers  (2025-09-11T03:06:22Z)
• Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector [24.390527651215944]
  Black-box adversarial attacks pose a particularly severe threat to Large Vision-Language Models (VLMs)<n>We propose a novel black-box targeted attack framework that leverages the projector.<n> Specifically, we utilize the widely adopted Querying Transformer (Q-Former) which transforms global image embeddings into fine-grained query outputs.
  arXiv  Detail & Related papers  (2025-08-19T11:23:09Z)
• MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction [8.503246256880612]
  We propose MaskGWM: a Generalizable driving World Model embodied with Video Mask reconstruction.<n>Our model contains two variants: MaskGWM-long, focusing on long-horizon prediction, and MaskGWM-mview, dedicated to multi-view generation.
  arXiv  Detail & Related papers  (2025-02-17T10:53:56Z)
• Knowing Where to Focus: Attention-Guided Alignment for Text-based Person Search [64.15205542003056]
  We introduce Attention-Guided Alignment (AGA) framework featuring two innovative components: Attention-Guided Mask (AGM) Modeling and Text Enrichment Module (TEM)<n>AGA achieves new state-of-the-art results with Rank-1 accuracy reaching 78.36%, 67.31%, and 67.4% on CUHK-PEDES, ICFG-PEDES, and RSTP, respectively.
  arXiv  Detail & Related papers  (2024-12-19T17:51:49Z)
• ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders [53.3185750528969]
  Masked AutoEncoders (MAE) have emerged as a robust self-supervised framework. We introduce a data-independent method, termed ColorMAE, which generates different binary mask patterns by filtering random noise. We demonstrate our strategy's superiority in downstream tasks compared to random masking.
  arXiv  Detail & Related papers  (2024-07-17T22:04:00Z)
• Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision [87.15580604023555]
  Unpair-Seg is a novel weakly-supervised open-vocabulary segmentation framework. It learns from unpaired image-mask and image-text pairs, which can be independently and efficiently collected. It achieves 14.6% and 19.5% mIoU on the ADE-847 and PASCAL Context-459 datasets.
  arXiv  Detail & Related papers  (2024-02-14T06:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.