World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning

• URL: http://arxiv.org/abs/2606.03603v1
• Date: Tue, 02 Jun 2026 13:07:49 GMT
• Title: World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning
• Abstract summary: We study how a model learns to invoke, verify, and integrate visual future simulation alongside abstract reasoning.<n>PF-OPSD uses ground-truth future videos and answers only as teacher-side privileged context to evaluate on-policy concrete-reasoning trajectories.
• Score: 67.93052893044603
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World models and multimodal large language models (MLLMs) provide complementary capabilities for predicting future outcomes from static visual observations. World models can generate concrete visual rollouts of possible futures, while MLLMs can reason abstractly over questions, goals, and rules. However, generated rollouts are stochastic and may be visually plausible but task-incorrect, making it necessary to determine when visual simulation is useful, whether a rollout is credible, and how it should influence the final answer. We formulate this problem as controlled concrete reasoning, where a model learns to invoke, verify, and integrate visual future simulation alongside abstract reasoning. To study this setting, we construct two human-verified benchmarks, VRQABench for controllable spatial lookahead and OpenWorldQA for open-domain physical prediction, and propose Privileged-Future On-Policy Self-Distillation (PF-OPSD). During training, PF-OPSD uses ground-truth future videos and answers only as teacher-side privileged context to evaluate on-policy concrete-reasoning trajectories, while the deployable student never observes true futures at test time. Experimental results show that PF-OPSD outperforms baseline by 10.6% and 10.9% on VRQABench and OpenWorldQA, respectively, while increasing robustness to noisy or conflicting rollouts. Our code and dataset are available at https://github.com/yczhou001/PF-OPSD.

Related papers

• WISE: World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Models [7.861070986808876]
  Post-training VLA policies typically rely on supervised fine-tuning with expensive and potentially unstable real-world exploration.<n>We introduce WISE (World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Models), a unified framework that coordinates when and how world-model imagination is used during policy refinement.
  arXiv  Detail & Related papers  (2026-09-03T11:17:57Z)
• Predicting Consequences and Reinforcing Navigation Policies with Latent World Models [31.865100130203317]
  We propose a compatibility prediction Latent World Model (LWM) for robot navigation.<n>Our key insight is that spatial proximity correlates with latent feature similarity, enabling action consequences to be evaluated directly in latent space.<n>Our approach significantly outperforms prior world model and imitation learning methods in prediction accuracy, policy learning, and real-world navigation performance.
  arXiv  Detail & Related papers  (2026-08-23T11:58:42Z)
• Learning POMDP World Models from Observations with Language-Model Priors [52.11151769796862]
  Partially-observable Markov decision processes (POMDPs) provide a flexible modeling class for internal world models.<n>We ask whether language-model priors can reduce costly interaction by leveraging prior knowledge.<n>We show that emphPinductor matches the performance and sample efficiency of LLM-based POMDP learning methods.
  arXiv  Detail & Related papers  (2026-05-13T16:18:15Z)
• ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving [19.081114003415863]
  We propose a unified understanding-and-generation framework to enable meaningful exploration and provide dense supervision.<n>We augment trajectory prediction with future RGB and depth image generation as dense world modeling objectives.<n>We incorporate this exploration signal into a safety-gated reward and optimize the policy.
  arXiv  Detail & Related papers  (2026-04-03T04:14:13Z)
• LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning [51.969318585152116]
  LatentPilot exploits future observations during training as a valuable data source to learn action-conditioned visual dynamics.<n>We propose a flywheel-style training mechanism that iteratively collects on-policy trajectories and retrains the model to better match the agent's behavior distribution.<n> Experiments on R2R-CE, RxR-CE, and R2R-PE benchmarks achieve new SOTA results, and real-robot tests across diverse environments demonstrate LatentPilot's superior understanding of environment-action dynamics in scene.
  arXiv  Detail & Related papers  (2026-03-31T02:21:59Z)
• World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy [55.03832008486675]
  World-VLA-Loop is a closed-loop framework for the joint refinement of world models and Vision-Language-Action (VLA) policies.<n>We propose a state-aware video world model that functions as a high-fidelity interactive simulator by jointly predicting future observations and reward signals.
  arXiv  Detail & Related papers  (2026-02-06T08:57:55Z)
• VFMF: World Modeling by Forecasting Vision Foundation Model Features [67.09340259579761]
  We introduce a generative forecaster that performs autoregressive flow matching in vision foundation models feature space.<n>We show that this latent information more effectively than previously used PCA-based alternatives, both for forecasting and other applications.<n>With matched architecture and compute, our method produces sharper and more accurate predictions than regression across all modalities.
  arXiv  Detail & Related papers  (2025-12-12T02:10:05Z)
• When Object-Centric World Models Meet Policy Learning: From Pixels to Policies, and Where It Breaks [24.669692812050645]
  We introduce a fully unsupervised, disentangled object-centric world model that learns object-level latents directly from pixels.<n>DLPWM achieves strong reconstruction and prediction performance, including robustness to several out-of-distribution (OOD) visual variations.<n>Our results suggest that, although object-centric perception supports robust visual modeling, achieving stable control requires mitigating latent drift.
  arXiv  Detail & Related papers  (2025-11-08T21:09:44Z)
• ACT-Bench: Towards Action Controllable World Models for Autonomous Driving [2.6749009435602122]
  World models have emerged as promising neural simulators for autonomous driving.<n>We develop an open-access evaluation framework, ACT-Bench, for quantifying action fidelity.<n>We demonstrate that the state-of-the-art model does not fully adhere to given instructions, while Terra achieves improved action fidelity.
  arXiv  Detail & Related papers  (2024-12-06T01:06:28Z)
• Predictive World Models from Real-World Partial Observations [66.80340484148931]
  We present a framework for learning a probabilistic predictive world model for real-world road environments. While prior methods require complete states as ground truth for learning, we present a novel sequential training method to allow HVAEs to learn to predict complete states from partially observed states only.
  arXiv  Detail & Related papers  (2023-01-12T02:07:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.