World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning

• URL: http://arxiv.org/abs/2604.26934v1
• Date: Wed, 29 Apr 2026 17:48:01 GMT
• Title: World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning
• Abstract summary: Vision-language models (VLMs) have shown strong performance on static visual understanding.<n>They still struggle with dynamic spatial reasoning that requires imagining how scenes evolve under egocentric motion.<n>We propose World2VLM, a training framework that distills spatial imagination from a generative world model into a vision-language model.
• Score: 18.658825757342544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) have shown strong performance on static visual understanding, yet they still struggle with dynamic spatial reasoning that requires imagining how scenes evolve under egocentric motion. Recent efforts address this limitation either by scaling spatial supervision with synthetic data or by coupling VLMs with world models at inference time. However, the former often lacks explicit modeling of motion-conditioned state transitions, while the latter incurs substantial computational overhead. In this work, we propose World2VLM, a training framework that distills spatial imagination from a generative world model into a vision-language model. Given an initial observation and a parameterized camera trajectory, we use a view-consistent world model to synthesize geometrically aligned future views and derive structured supervision for both forward (action-to-outcome) and inverse (outcome-to-action) spatial reasoning. We post-train the VLM with a two-stage recipe on a compact dataset generated by this pipeline and evaluate it on multiple spatial reasoning benchmarks. World2VLM delivers consistent improvements over the base model across diverse benchmarks, including SAT-Real, SAT-Synthesized, VSI-Bench, and MindCube. It also outperforms the test-time world-model-coupled methods while eliminating the need for expensive inference-time generation. Our results suggest that world models can serve not only as inference-time tools, but also as effective training-time teachers, enabling VLMs to internalize spatial imagination in a scalable and efficient manner.

Related papers

• GeoWorld-VLM: Geometry from World Models for Vision-Language Models [10.86505613923278]
  Modern Vision-Language Models (VLMs) achieve strong semantic recognition, yet remain brittle on elementary spatial relations.<n>We introduce GeoWorld-VLM, a VLM-side distillation framework that transfers geometric structure from frozen video world models intoVLMs.<n>GeoWorld-VLM fine-tunes only the image encoder and multimodal projector, aligning post-projector image features with intermediate world-model representations while leaving the main backbone frozen.
  arXiv  Detail & Related papers  (2026-05-15T23:52:11Z)
• Learning Vision-Language-Action World Models for Autonomous Driving [15.103497388527943]
  Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving.<n>VLA models often lack explicit modeling of temporal dynamics and global world consistency.<n>We present VLA-World, a simple yet effective VLA world model that unifies predictive imagination with reflective reasoning.
  arXiv  Detail & Related papers  (2026-04-10T07:38:05Z)
• Causal World Modeling for Robot Control [56.31803788587547]
  Video world models provide the ability to imagine the near future by understanding the causality between actions and visual dynamics.<n>We introduce LingBot-VA, an autoregressive diffusion framework that learns frame prediction and policy execution simultaneously.<n>We evaluate our model on both simulation benchmarks and real-world scenarios, where it shows significant promise in long-horizon manipulation, data efficiency in post-training, and strong generalizability to novel configurations.
  arXiv  Detail & Related papers  (2026-01-29T17:07:43Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model [53.555353366322464]
  We present TeleWorld, a real-time multimodal 4D world modeling framework that unifies video generation, dynamic scene reconstruction, and long-term world memory within a closed-loop system.<n>Our approach achieves seamless integration of dynamic object modeling and static scene representation within a unified 4D framework, advancing world models toward practical, interactive, and computationally accessible synthesis systems.
  arXiv  Detail & Related papers  (2025-12-31T18:31:46Z)
• VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation [23.86958300272144]
  We introduce VDAWorld, a framework where a Vision-Language Model (VLM) acts as an intelligent agent to orchestrate this process.<n>The VLM autonomously constructs a grounded (2D or 3D) scene representation by selecting from a suite of vision tools.<n>It can then infer latent dynamics from the static scene to predict plausible future states.
  arXiv  Detail & Related papers  (2025-12-11T19:21:47Z)
• Can World Models Benefit VLMs for World Dynamics? [59.73433292793044]
  We investigate the capabilities when world model priors are transferred into Vision-Language Models.<n>We name our best-performing variant Dynamic Vision Aligner (DyVA)<n>We find DyVA to surpass both open-source and proprietary baselines, achieving state-of-the-art or comparable performance.
  arXiv  Detail & Related papers  (2025-10-01T13:07:05Z)
• Learning Primitive Embodied World Models: Towards Scalable Robotic Learning [50.32986780156215]
  We propose a novel paradigm for world modeling--Primitive Embodied World Models (PEWM)<n>By restricting video generation to fixed short horizons, our approach enables fine-grained alignment between linguistic concepts and visual representations of robotic actions.<n>Our framework bridges the gap between fine-grained physical interaction and high-level reasoning, paving the way toward scalable, interpretable, and general-purpose embodied intelligence.
  arXiv  Detail & Related papers  (2025-08-28T14:31:48Z)
• MindJourney: Test-Time Scaling with World Models for Spatial Reasoning [97.61985090279961]
  We propose MindJourney, a test-time scaling framework for vision-language models.<n>We show that MindJourney achieves over an average 7.7% performance boost on the representative spatial reasoning benchmark SAT.<n>Our method also improves upon the test-time inference VLMs trained through reinforcement learning.
  arXiv  Detail & Related papers  (2025-07-16T17:59:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.