Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World

• URL: http://arxiv.org/abs/2603.12746v1
• Date: Fri, 13 Mar 2026 07:42:16 GMT
• Title: Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World
• Abstract summary: Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time.<n>We introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets.<n>We find that existing models cannot simultaneously maintain strong performance in both-temporal reasoning and dynamic object grounding.
• Score: 49.80040477190479
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time, constituting a dynamic 4D reality (spatial with temporal dimension). While current Multimodal Large Language Models (MLLMs) excel in static visual understanding, can they also be adept at "thinking in dynamics", i.e., perceive, track and reason about spatio-temporal dynamics in evolving scenes? To systematically assess their spatio-temporal reasoning and localized dynamics perception capabilities, we introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets, enabling robust and scalable evaluation of spatio-temporal understanding. Through multi-stage filtering from massive 2D and 4D data sources, Dyn-Bench provides a high-quality collection of dynamic scenes, comprising 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding pairs. We probe general, spatial and region-level MLLMs to express how they think in dynamics both linguistically and visually, and find that existing models cannot simultaneously maintain strong performance in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Notably, conventional prompting strategies (e.g., chain-of-thought or caption-based hints) provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and Spatio-Temporal Textual Cognitive Map (ST-TCM), significantly enhance MLLMs' dynamics perception and spatio-temporal reasoning in the physical 4D world. Code and benchmark are available at https://dyn-bench.github.io/.

Related papers

• DynTrace: Tracking Dynamic Object Evidence for 4D Spatio-Temporal Reasoning in MLLMs [41.4243802276804]
  4D-temporal reasoning is essential for understanding dynamic worlds and enabling embodied interaction.<n>Current models rely on sparse frame-level observations, fragmenting continuous dynamic cues.<n>We propose DynTrace, a training-free framework for 4D-temporal reasoning with two complementary components.
  arXiv  Detail & Related papers  (2026-07-14T08:35:07Z)
• 4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding [31.082079260882896]
  We present 4DThinker, the first framework that enables vision-language models to "think with 4D"<n>We first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos.<n>We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly verbose textual tokens and 4D latents to ground the model in dynamic visual semantics.
  arXiv  Detail & Related papers  (2026-05-07T10:48:46Z)
• PanopticQuery: Unified Query-Time Reasoning for 4D Scenes [53.672906752290665]
  We introduce PanopticQuery, a framework for unified query-time reasoning in 4D scenes.<n>Our approach builds on 4D Gaussian Splatting for high-fidelity dynamic reconstruction.<n>We present Panoptic-L4D, a new benchmark for language-based querying in dynamic scenes.
  arXiv  Detail & Related papers  (2026-04-07T09:40:05Z)
• TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model [53.555353366322464]
  We present TeleWorld, a real-time multimodal 4D world modeling framework that unifies video generation, dynamic scene reconstruction, and long-term world memory within a closed-loop system.<n>Our approach achieves seamless integration of dynamic object modeling and static scene representation within a unified 4D framework, advancing world models toward practical, interactive, and computationally accessible synthesis systems.
  arXiv  Detail & Related papers  (2025-12-31T18:31:46Z)
• DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling [67.95038177144554]
  We introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video.<n>We employ vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic captions.<n> DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos.
  arXiv  Detail & Related papers  (2025-12-02T18:24:27Z)
• MASS: Motion-Aware Spatial-Temporal Grounding for Physics Reasoning and Comprehension in Vision-Language Models [45.450035386882824]
  Vision Language Models (VLMs) perform well on standard video tasks but struggle with physics-driven reasoning involving motion dynamics and spatial interactions.<n>We present an approach that addresses this gap by translating physical-world context cues into interpretable representations aligned with VLMs' perception, comprehension, and reasoning.
  arXiv  Detail & Related papers  (2025-11-23T09:43:44Z)
• Understanding Dynamic Scenes in Ego Centric 4D Point Clouds [7.004204907286336]
  EgoDynamic4D is a novel QA benchmark on highly dynamic scenes.<n>We design 12 dynamic QA tasks covering agent motion, human-object interaction prediction, relation, understanding trajectory, temporal-causal reasoning, with fine-grained, explicit metrics.<n>Our method consistently outperforms baselines, validating the effectiveness of multimodal temporal modeling for ego-centric dynamic scene understanding.
  arXiv  Detail & Related papers  (2025-08-10T09:08:04Z)
• VLM4D: Towards Spatiotemporal Awareness in Vision Language Models [66.833085504228]
  We introduce V4DLM, the first benchmark specifically designed to evaluate visual language models (VLMs)<n>Our benchmark comprises diverse real-world and synthetic videos accompanied by carefully curated question-answer pairs.<n>We identify significant performance gaps compared to human baselines, highlighting fundamental deficiencies in existing models.
  arXiv  Detail & Related papers  (2025-08-04T06:06:06Z)
• EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting [95.44545809256473]
  EgoGaussian is a method capable of simultaneously reconstructing 3D scenes and dynamically tracking 3D object motion from RGB egocentric input alone. We show significant improvements in terms of both dynamic object and background reconstruction quality compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2024-06-28T10:39:36Z)
• DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization [67.85434518679382]
  We present DynaVol, a 3D scene generative model that unifies geometric structures and object-centric learning. The key idea is to perform object-centric voxelization to capture the 3D nature of the scene. voxel features evolve over time through a canonical-space deformation function, forming the basis for global representation learning.
  arXiv  Detail & Related papers  (2023-04-30T05:29:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.