4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding

• URL: http://arxiv.org/abs/2605.05997v1
• Date: Thu, 07 May 2026 10:48:46 GMT
• Title: 4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding
• Abstract summary: We present 4DThinker, the first framework that enables vision-language models to "think with 4D"<n>We first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos.<n>We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly verbose textual tokens and 4D latents to ground the model in dynamic visual semantics.
• Score: 31.082079260882896
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dynamic spatial reasoning from monocular video is essential for bridging visual intelligence and the physical world, yet remains challenging for vision-language models (VLMs). Prior approaches either verbalize spatial-temporal reasoning entirely as text, which is inherently verbose and imprecise for complex dynamics, or rely on external geometric modules that increase inference complexity without fostering intrinsic model capability. In this paper, we present 4DThinker, the first framework that enables VLMs to "think with 4D" through dynamic latent mental imagery, i.e., internally simulating how scenes evolve within the continuous hidden space. Specifically, we first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos. We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly supervises textual tokens and 4D latents to ground the model in dynamic visual semantics. Building on this, 4D Reinforcement Learning (4DRL) further tackles complex reasoning tasks via outcome-based rewards, restricting policy gradients to text tokens to ensure stable optimization. Extensive experiments across multiple dynamic spatial reasoning benchmarks demonstrate that 4DThinker consistently outperforms strong baselines and offers a new perspective toward 4D reasoning in VLMs. Our code is available at https://github.com/zhangquanchen/4DThinker.

Related papers

• CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes [2.5603340362916938]
  We present CL4D, the first foundational 4D vision encoder that directly operates on dynamic point clouds.<n>We introduce 4DVLM, a 4D vision-language model that conditions language generation on dynamic geometric representations.<n> CL4D achieves state-of-the-art performance, with improvements of approximately 16.75% over prior methods.
  arXiv  Detail & Related papers  (2026-08-19T09:37:15Z)
• PanopticQuery: Unified Query-Time Reasoning for 4D Scenes [53.672906752290665]
  We introduce PanopticQuery, a framework for unified query-time reasoning in 4D scenes.<n>Our approach builds on 4D Gaussian Splatting for high-fidelity dynamic reconstruction.<n>We present Panoptic-L4D, a new benchmark for language-based querying in dynamic scenes.
  arXiv  Detail & Related papers  (2026-04-07T09:40:05Z)
• 4C4D: 4 Camera 4D Gaussian Splatting [56.926598190499156]
  This paper tackles the challenge of recovering 4D dynamic scenes from videos captured by as few as four portable cameras.<n>We propose textbf4C4D, a novel framework that enables high-fidelity 4D Gaussian Splatting from video captures of extremely sparse cameras.
  arXiv  Detail & Related papers  (2026-04-05T11:09:23Z)
• Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World [49.80040477190479]
  Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time.<n>We introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets.<n>We find that existing models cannot simultaneously maintain strong performance in both-temporal reasoning and dynamic object grounding.
  arXiv  Detail & Related papers  (2026-03-13T07:42:16Z)
• Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models [79.18306680174011]
  DSR Suite bridges gap across aspects of dataset, benchmark and model.<n>We propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR.<n>The pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories.
  arXiv  Detail & Related papers  (2025-12-23T17:56:36Z)
• Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation [61.60600246983274]
  Existing 3D and 4D approaches typically embed scene geometry into autogressive model for semantic understanding and diffusion model for content generation.<n>We propose Uni4D-LLM, the first unified VLM framework withtemporal awareness for 4D scene understanding and generation.
  arXiv  Detail & Related papers  (2025-09-28T12:06:54Z)
• 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives [115.67081491747943]
  Dynamic 3D scene representation and novel view synthesis are crucial for enabling AR/VR and metaverse applications.<n>We reformulate the reconstruction of a time-varying 3D scene as approximating its underlying 4D volume.<n>We derive several compact variants that effectively reduce the memory footprint to address its storage bottleneck.
  arXiv  Detail & Related papers  (2024-12-30T05:30:26Z)
• Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering [23.04702935216809]
  We introduce DynSuperCLEVR, the first video question answering dataset that focuses on language understanding of the dynamic properties of 3D objects.<n>We generate three types of questions, including factual queries, future predictions, and counterfactual reasoning.<n>Our method first estimates the 4D world states with a 3D generative model powered by physical priors, and then uses neural symbolic reasoning to answer the questions based on the 4D world states.
  arXiv  Detail & Related papers  (2024-06-02T05:51:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.