PanopticQuery: Unified Query-Time Reasoning for 4D Scenes

• URL: http://arxiv.org/abs/2604.05638v1
• Date: Tue, 07 Apr 2026 09:40:05 GMT
• Title: PanopticQuery: Unified Query-Time Reasoning for 4D Scenes
• Abstract summary: We introduce PanopticQuery, a framework for unified query-time reasoning in 4D scenes.<n>Our approach builds on 4D Gaussian Splatting for high-fidelity dynamic reconstruction.<n>We present Panoptic-L4D, a new benchmark for language-based querying in dynamic scenes.
• Score: 53.672906752290665
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding dynamic 4D environments through natural language queries requires not only accurate scene reconstruction but also robust semantic grounding across space, time, and viewpoints. While recent methods using neural representations have advanced 4D reconstruction, they remain limited in contextual reasoning, especially for complex semantics such as interactions, temporal actions, and spatial relations. A key challenge lies in transforming noisy, view-dependent predictions into globally consistent 4D interpretations. We introduce PanopticQuery, a framework for unified query-time reasoning in 4D scenes. Our approach builds on 4D Gaussian Splatting for high-fidelity dynamic reconstruction and introduces a multi-view semantic consensus mechanism that grounds natural language queries by aggregating 2D semantic predictions across multiple views and time frames. This process filters inconsistent outputs, enforces geometric consistency, and lifts 2D semantics into structured 4D groundings via neural field optimization. To support evaluation, we present Panoptic-L4D, a new benchmark for language-based querying in dynamic scenes. Experiments demonstrate that PanopticQuery sets a new state of the art on complex language queries, effectively handling attributes, actions, spatial relationships, and multi-object interactions. A video demonstration is available in the supplementary materials.

Related papers

• IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer [64.5765465030666]
  We propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding.<n> IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity.<n> Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines.
  arXiv  Detail & Related papers  (2026-07-21T16:00:01Z)
• 4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding [31.082079260882896]
  We present 4DThinker, the first framework that enables vision-language models to "think with 4D"<n>We first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos.<n>We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly verbose textual tokens and 4D latents to ground the model in dynamic visual semantics.
  arXiv  Detail & Related papers  (2026-05-07T10:48:46Z)
• Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World [49.80040477190479]
  Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time.<n>We introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets.<n>We find that existing models cannot simultaneously maintain strong performance in both-temporal reasoning and dynamic object grounding.
  arXiv  Detail & Related papers  (2026-03-13T07:42:16Z)
• SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning [11.93789125154006]
  We propose a framework for unified 4D scene understanding that integrates VLM-derived semantics with point cloud geometry and temporal consistency.<n>SNOW processes synchronized 3D point clouds, using HDBSCAN clustering to generate segmentation proposals.<n> Experiments on a diverse set of benchmarks demonstrate that SNOW enables precise 4D scene understanding and spatially grounded inference.
  arXiv  Detail & Related papers  (2025-12-18T12:27:06Z)
• R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space [11.93789125154006]
  R4 is a training-free framework for retrieval-augmented reasoning in 4D space.<n>R4 constructs a 4D database by anchoring object-level semantic descriptions in metric space and time.<n>At inference, natural language queries are decomposed into semantic, spatial, and temporal keys to retrieve relevant observations.
  arXiv  Detail & Related papers  (2025-12-17T20:08:32Z)
• 4DLangVGGT: 4D Language-Visual Geometry Grounded Transformer [40.29321632546414]
  4DVGGT is the first Transformer-based feed-forward unified framework for 4D language grounding.<n>It integrates geometric perception and language alignment within a single architecture.<n>It can be jointly trained across multiple dynamic scenes and directly applied during inference.
  arXiv  Detail & Related papers  (2025-12-04T18:15:27Z)
• RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregation [10.067978300536486]
  We develop a zero-shot framework that seamlessly integrates GPU-accelerated geometric reconstruction with open-vocabulary vision-language models.<n>Our training-free system achieves superior performance through incremental processing and unified geometric-semantic updates.
  arXiv  Detail & Related papers  (2025-05-21T11:07:25Z)
• LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding [55.81291976637705]
  We propose a general LMM framework with atemporal prompt for visual representation 4D scene understanding.<n>The prompt is generated by encoding 3D position and 1D time into dynamic-aware 4D coordinate embedding.<n>Experiments have been conducted to demonstrate the effectiveness of our method across different tasks in 4D scene understanding.
  arXiv  Detail & Related papers  (2025-05-18T06:18:57Z)
• 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models [58.80200897869225]
  We propose 4D LangSplat, which learns 4D language fields to handle time-agnostic or time-sensitive open-vocabulary queries in dynamic scenes efficiently.<n>4D LangSplat bypasses learning the language field from vision features and instead learns directly from text generated from object-wise video captions.<n>Our results demonstrate that 4D LangSplat attains precise and efficient results for both time-sensitive and time-agnostic open-vocabulary queries.
  arXiv  Detail & Related papers  (2025-03-13T14:58:22Z)
• 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives [115.67081491747943]
  Dynamic 3D scene representation and novel view synthesis are crucial for enabling AR/VR and metaverse applications.<n>We reformulate the reconstruction of a time-varying 3D scene as approximating its underlying 4D volume.<n>We derive several compact variants that effectively reduce the memory footprint to address its storage bottleneck.
  arXiv  Detail & Related papers  (2024-12-30T05:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.