R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space

• URL: http://arxiv.org/abs/2512.15940v1
• Date: Wed, 17 Dec 2025 20:08:32 GMT
• Title: R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space
• Abstract summary: R4 is a training-free framework for retrieval-augmented reasoning in 4D space.<n>R4 constructs a 4D database by anchoring object-level semantic descriptions in metric space and time.<n>At inference, natural language queries are decomposed into semantic, spatial, and temporal keys to retrieve relevant observations.
• Score: 11.93789125154006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans perceive and reason about their surroundings in four dimensions by building persistent, structured internal representations that encode semantic meaning, spatial layout, and temporal dynamics. These multimodal memories enable them to recall past events, infer unobserved states, and integrate new information into context-dependent reasoning. Inspired by this capability, we introduce R4, a training-free framework for retrieval-augmented reasoning in 4D spatio-temporal space that equips vision-language models (VLMs) with structured, lifelong memory. R4 continuously constructs a 4D knowledge database by anchoring object-level semantic descriptions in metric space and time, yielding a persistent world model that can be shared across agents. At inference, natural language queries are decomposed into semantic, spatial, and temporal keys to retrieve relevant observations, which are integrated into the VLM's reasoning. Unlike classical retrieval-augmented generation methods, retrieval in R4 operates directly in 4D space, enabling episodic and collaborative reasoning without training. Experiments on embodied question answering and navigation benchmarks demonstrate that R4 substantially improves retrieval and reasoning over spatio-temporal information compared to baselines, advancing a new paradigm for embodied 4D reasoning in dynamic environments.

Related papers

• 4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding [31.082079260882896]
  We present 4DThinker, the first framework that enables vision-language models to "think with 4D"<n>We first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos.<n>We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly verbose textual tokens and 4D latents to ground the model in dynamic visual semantics.
  arXiv  Detail & Related papers  (2026-05-07T10:48:46Z)
• DINO_4D: Semantic-Aware 4D Reconstruction [0.6457144394797564]
  We present DINO_4D, introducing frozen DINOv3 features as structural priors, injecting semantic awareness into the reconstruction process.<n>Experiments on the Point Odyssey and TUM-Dynamics benchmarks demonstrate that our method maintains the linear time complexity $O(T)$ of its predecessors.<n>DINO_4D establishes a new paradigm for constructing 4D World Models that possess both geometric precision and semantic understanding.
  arXiv  Detail & Related papers  (2026-04-10T20:07:14Z)
• PanopticQuery: Unified Query-Time Reasoning for 4D Scenes [53.672906752290665]
  We introduce PanopticQuery, a framework for unified query-time reasoning in 4D scenes.<n>Our approach builds on 4D Gaussian Splatting for high-fidelity dynamic reconstruction.<n>We present Panoptic-L4D, a new benchmark for language-based querying in dynamic scenes.
  arXiv  Detail & Related papers  (2026-04-07T09:40:05Z)
• Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World [49.80040477190479]
  Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time.<n>We introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets.<n>We find that existing models cannot simultaneously maintain strong performance in both-temporal reasoning and dynamic object grounding.
  arXiv  Detail & Related papers  (2026-03-13T07:42:16Z)
• MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence [50.11889361459544]
  Humans are born with vision-based 4D spatial-temporal intelligence.<n>Despite its importance, this capability remains a significant bottleneck for current large language models (MLLMs)
  arXiv  Detail & Related papers  (2026-02-28T07:23:36Z)
• ReScene4D: Temporally Consistent Semantic Instance Segmentation of Evolving Indoor 3D Scenes [11.119542051581917]
  We introduce and formalize the task of temporally sparse 4D indoor semantic instance segmentation (SIS)<n>We propose ReScene4D, a novel method that adapts 3DSIS architectures for 4DSIS without needing dense observations.<n>To evaluate this task, we define a new metric, t-mAP, that extends mAP to reward temporal identity consistency.
  arXiv  Detail & Related papers  (2026-01-16T18:45:19Z)
• Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmark [68.52969403388082]
  Humans naturally possess 4D spatial intelligence, supporting a broad spectrum of reasoning abilities.<n>To what extent can Multimodal Large Language Models (MLLMs) achieve human-level 4D spatial intelligence?<n>We present Spatial4D-Bench, a versatile 4D spatial intelligence benchmark designed to assess the 4D spatial reasoning abilities of MLLMs.
  arXiv  Detail & Related papers  (2025-12-31T19:56:51Z)
• Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models [79.18306680174011]
  DSR Suite bridges gap across aspects of dataset, benchmark and model.<n>We propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR.<n>The pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories.
  arXiv  Detail & Related papers  (2025-12-23T17:56:36Z)
• SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning [11.93789125154006]
  We propose a framework for unified 4D scene understanding that integrates VLM-derived semantics with point cloud geometry and temporal consistency.<n>SNOW processes synchronized 3D point clouds, using HDBSCAN clustering to generate segmentation proposals.<n> Experiments on a diverse set of benchmarks demonstrate that SNOW enables precise 4D scene understanding and spatially grounded inference.
  arXiv  Detail & Related papers  (2025-12-18T12:27:06Z)
• STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence [81.94084852268468]
  We formalize audio 4D intelligence that is defined as reasoning over sound dynamics in time and 3D space.<n> STAR-Bench combines a Foundational Acoustic Perception setting with a Holistic Spatio-Temporal Reasoning setting.<n>Our data curation pipeline uses two methods to ensure high-quality samples.
  arXiv  Detail & Related papers  (2025-10-28T17:50:34Z)
• Understanding Dynamic Scenes in Ego Centric 4D Point Clouds [7.004204907286336]
  EgoDynamic4D is a novel QA benchmark on highly dynamic scenes.<n>We design 12 dynamic QA tasks covering agent motion, human-object interaction prediction, relation, understanding trajectory, temporal-causal reasoning, with fine-grained, explicit metrics.<n>Our method consistently outperforms baselines, validating the effectiveness of multimodal temporal modeling for ego-centric dynamic scene understanding.
  arXiv  Detail & Related papers  (2025-08-10T09:08:04Z)
• Reconstructing 4D Spatial Intelligence: A Survey [57.8684548664209]
  Reconstructing 4D spatial intelligence from visual observations has long been a central yet challenging task in computer vision.<n>We present a new perspective that organizes existing methods into five progressive levels of 4D spatial intelligence.
  arXiv  Detail & Related papers  (2025-07-28T17:59:02Z)
• LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding [55.81291976637705]
  We propose a general LMM framework with atemporal prompt for visual representation 4D scene understanding.<n>The prompt is generated by encoding 3D position and 1D time into dynamic-aware 4D coordinate embedding.<n>Experiments have been conducted to demonstrate the effectiveness of our method across different tasks in 4D scene understanding.
  arXiv  Detail & Related papers  (2025-05-18T06:18:57Z)
• ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos [8.279721795956124]
  Humans excel at spatial-temporal reasoning, effortlessly interpreting dynamic visual events from an egocentric viewpoint.<n>This paper explores multimodal spatial-temporal reasoning from an egocentric perspective, aiming to equip MLLMs with human-like reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-16T15:24:11Z)
• 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives [115.67081491747943]
  Dynamic 3D scene representation and novel view synthesis are crucial for enabling AR/VR and metaverse applications.<n>We reformulate the reconstruction of a time-varying 3D scene as approximating its underlying 4D volume.<n>We derive several compact variants that effectively reduce the memory footprint to address its storage bottleneck.
  arXiv  Detail & Related papers  (2024-12-30T05:30:26Z)
• Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering [23.04702935216809]
  We introduce DynSuperCLEVR, the first video question answering dataset that focuses on language understanding of the dynamic properties of 3D objects.<n>We generate three types of questions, including factual queries, future predictions, and counterfactual reasoning.<n>Our method first estimates the 4D world states with a 3D generative model powered by physical priors, and then uses neural symbolic reasoning to answer the questions based on the 4D world states.
  arXiv  Detail & Related papers  (2024-06-02T05:51:15Z)
• LoRD: Local 4D Implicit Representation for High-Fidelity Dynamic Human Modeling [69.56581851211841]
  We propose a novel Local 4D implicit Representation for Dynamic clothed human, named LoRD. Our key insight is to encourage the network to learn the latent codes of local part-level representation. LoRD has strong capability for representing 4D human, and outperforms state-of-the-art methods on practical applications.
  arXiv  Detail & Related papers  (2022-08-18T03:49:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.