Dreaming when Necessary: Advancing World Action Models with Adaptive Multi-Modal Reasoning

• URL: http://arxiv.org/abs/2606.07089v1
• Date: Fri, 05 Jun 2026 09:35:48 GMT
• Title: Dreaming when Necessary: Advancing World Action Models with Adaptive Multi-Modal Reasoning
• Abstract summary: World Action Models (WAMs) offer a promising approach to embodied intelligence.<n>Existing methods rely heavily on video prediction as action priors and lack adaptive multimodal reasoning.<n>We propose textbfAdaWAM, a world action model with adaptive multimodal reasoning abilities.
• Score: 12.751695880459053
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: World Action Models (WAMs) offer a promising approach to embodied intelligence, yet existing methods rely heavily on video prediction as action priors and lack adaptive multimodal reasoning, limiting their effectiveness on long-horizon, complex tasks. We observe that WAMs require different multimodal reasoning modes under different execution contexts: textual reasoning is essential during task transitions to guide high-level action prediction, while visual reasoning is critical during fine-grained manipulation for precise control. Motivated by this observation, we propose \textbf{AdaWAM}, a world action model with adaptive multimodal reasoning abilities. AdaWAM integrates a lightweight dynamic router that autonomously triggers textual or visual reasoning as needed during task execution. Experiments on both simulated and real-world embodied tasks show that AdaWAM substantially improves inference efficiency while outperforming state-of-the-art embodied policies. Codes and demos are available at: https://adawam.github.io/.

Related papers

• HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models [58.191567345416836]
  World Action Models (WAMs) have emerged as a promising paradigm for robot control by modeling physical dynamics.<n>HarmoWAM fully leverages a world model to unify predictive and reactive control, enabling both general transit and precise manipulation.<n>We construct three training-unseen test environments across six real-world robotic tasks, covering variations in background, position, and object semantics.
  arXiv  Detail & Related papers  (2026-05-11T17:59:56Z)
• Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization [89.68681087743876]
  Multimodal Large Language Models (MLLMs) have incentivized models to think with images'' by actively invoking visual tools during multi-turn reasoning.<n>The common Reinforcement Learning practice of relying on outcome-based rewards ignores the fact that textual plausibility often masks executive failure.<n>We introduce Multimodal Agentic Policy Optimization (MAPO), bridging the gap between textual reasoning and visual actions generated by models within their Multimodal Chain-of-Thought.
  arXiv  Detail & Related papers  (2026-04-08T07:48:07Z)
• Perception-Aware Multimodal Spatial Reasoning from Monocular Images [57.42071289037214]
  spatial reasoning from monocular images is essential for autonomous driving.<n>Current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception.<n>We propose a perception-aware multimodal reasoning framework that equipsVLMs with explicit object-centric grounding ability.
  arXiv  Detail & Related papers  (2026-03-07T02:05:12Z)
• Omni-AutoThink: Adaptive Multimodal Reasoning via Reinforcement Learning [57.96134674544638]
  We propose a novel adaptive reasoning framework that dynamically adjusts the model's reasoning depth according to task difficulty.<n>Our framework comprises two stages: (1) an Adaptive Supervised Fine-Tuning stage, which endows the Omni model with fundamental reasoning capability using large-scale reasoning-augmented data, and (2) an Adaptive Reinforcement Learning stage, which optimize reasoning behaviors based on task complexity and reward feedback.
  arXiv  Detail & Related papers  (2025-12-03T13:33:28Z)
• UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Model [19.343780691204792]
  Vision-and-Language Navigation (VLN) requires agents to autonomously navigate complex environments via visual images and natural language instruction.<n>Recent research on enhancing language-guided navigation reasoning using pre-trained large language models (LLMs) has shown promising prospects.<n>We introduce UNeMo, a novel framework designed for the collaborative optimization of visual state reasoning and navigational decision-making.
  arXiv  Detail & Related papers  (2025-11-24T07:31:58Z)
• MultiModal Action Conditioned Video Generation [30.4362754101642]
  General-purpose household robots require real-time fine motor control to handle delicate tasks and urgent situations.<n>We consider senses of proprioception, kinesthesia, force haptics, and muscle activation.<n>Experiments show that incorporating multimodal senses improves simulation accuracy and reduces temporal drift.
  arXiv  Detail & Related papers  (2025-10-02T17:57:06Z)
• Dynamic Prompt Fusion for Multi-Task and Cross-Domain Adaptation in LLMs [2.852258765983155]
  This study introduces a unified multi-task learning framework with dynamic prompt scheduling mechanism.<n>It enhances the model's ability to capture semantic differences across tasks.<n>It incorporates an automatic learning strategy for scheduling weights, which effectively mitigates task interference and negative transfer.
  arXiv  Detail & Related papers  (2025-09-09T23:42:16Z)
• ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning [47.27336786187929]
  Vision-language-action (VLA) reasoning tasks require agents to interpret multimodal instructions, perform long-horizon planning, and act adaptively in dynamic environments.<n>Existing approaches typically train VLA models in an end-to-end fashion, directly mapping inputs to actions without explicit reasoning.<n>We propose ThinkAct, a dual-system framework that bridges high-level reasoning with low-level action execution via reinforced visual latent planning.
  arXiv  Detail & Related papers  (2025-07-22T17:59:46Z)
• EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Ability [11.721839449847472]
  We introduce MM-Escape, a benchmark for investigating multimodal reasoning.<n> MM-Escape emphasizes intermediate model behaviors alongside final task completion.<n>Extensive experiments show that MLLMs, regardless of scale, can successfully complete the simplest room escape tasks.<n>We observe that performance bottlenecks vary across models, revealing distinct failure modes and limitations in their multimodal reasoning abilities.
  arXiv  Detail & Related papers  (2025-03-13T04:48:43Z)
• Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment [58.94611347128066]
  multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals.<n>Recent studies either develop tool-using or unify specific visual tasks into the autoregressive framework, often at the expense of overall multimodal performance.<n>We propose Task Preference Optimization (TPO), a novel method that utilizes differentiable task preferences derived from typical fine-grained visual tasks.
  arXiv  Detail & Related papers  (2024-12-26T18:56:05Z)
• Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving [103.745551954983]
  In this paper, we investigate the transfer performance of various types of self-supervised methods, including MoCo and SimCLR, on three downstream tasks. We find that their performances are sub-optimal or even lag far behind the single-task baseline. We propose a simple yet effective pretrain-adapt-finetune paradigm for general multi-task training.
  arXiv  Detail & Related papers  (2022-09-19T12:15:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.