See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning

• URL: http://arxiv.org/abs/2603.23864v1
• Date: Wed, 25 Mar 2026 02:45:03 GMT
• Title: See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning
• Abstract summary: We introduce S3-Bench, a benchmark suite for streaming spatial question answering with active exploration.<n>S3-Bench adopts a dual-domain design, combining a scalable simulator with controllable trajectories and exploration actions.<n>Our approach yields improvements of 8.8% and 13.3% on the simulated and real splits of S3-Eval, respectively.
• Score: 41.278682336342165
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spatial understanding is fundamental for embodied agents, yet most spatial VLMs and benchmarks remain offline-evaluating post-hoc QA over pre-recorded inputs and overlooking two crucial deployment-critical requirements: long-horizon streaming inference and active perception when the current view is insufficient. To address this gap, we introduce S3-Bench, a benchmark suite for streaming spatial question answering with active exploration, where queries are temporally grounded to specific timestamps and must be answered using only observations available up to that moment. S3-Bench adopts a dual-domain design, combining a scalable simulator with controllable trajectories and exploration actions, and real-world streaming videos that capture practical sensing artifacts for rigorous generalization evaluation. Overall, it spans 10K+ scenes and 26K+ trajectories, with dedicated training (S3-Train) and evaluation (S3-Eval) splits. We further propose AMF-VLM, which supports streaming spatial reasoning under bounded computing via (i) memory folding, which compresses long-horizon observations into compact structured memory, and (ii) active exploration, which outputs explicit actions (e.g. move/rotate/scan) to acquire missing evidence before answering. Extensive experiments demonstrate that, compared to models using identical training data, our approach yields improvements of 8.8% and 13.3% on the simulated and real splits of S3-Eval, respectively, while maintaining competitive transferability to standard spatial benchmarks.

Related papers

• Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding [54.840531495584095]
  We propose a dynamic visual selection framework named EviSelect grounded in the target MLLM's internal attention evidence.<n>Our method efficiently approximates maps of the target MLLM using highly compressed visual inputs and sparse attention.<n>EviSelect achieves superior performance compared to existing methods while reducing selected visual tokens by about 50% and achieving a 3.9x end-to-end speedup.
  arXiv  Detail & Related papers  (2026-08-06T09:15:43Z)
• LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence [107.63317552620231]
  We present LLaVA-OneVision-2 (LLaVA-OV-2), the most capable vision-OneVision model to date.<n>It achieves superior performance across a broad range of multimodal benchmarks.<n>A standout capability is its unified perception across video understanding, temporal grounding, spatial grounding, and manipulation-language reasoning.
  arXiv  Detail & Related papers  (2026-05-25T15:54:04Z)
• Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence [78.1406635199656]
  Holi-Spatial is the first fully automated, large-scale, spatially-aware multimodal dataset constructed from raw video inputs without human intervention.<n>Holi-Spatial-4M is the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs.
  arXiv  Detail & Related papers  (2026-03-08T14:49:20Z)
• Real-Time 3D Object Detection with Inference-Aligned Learning [20.94871746774727]
  Real-time 3D object detection from point clouds is essential for dynamic scene understanding in applications such as augmented reality, robotics and navigation.<n>We introduce a novel Spatial-prioritized and Rank-aware 3D object detection framework to bridge the gap between how detectors are trained and how they are evaluated.
  arXiv  Detail & Related papers  (2025-11-20T08:27:00Z)
• Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence [70.2803680525165]
  We introduce Open-o3 Video, a non-agent framework that integrates explicit evidence into video reasoning.<n>The model highlights key objects and bounding boxes alongside its answers, allowing reasoning to be grounded in concrete visual observations.<n>On V-STAR benchmark, Open-o3 Video achieves state-of-the-art performance, raising mAM by 14.4% and mL timestamp by 24.2%.
  arXiv  Detail & Related papers  (2025-10-23T14:05:56Z)
• OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding [50.72259772580637]
  We introduce OST-Bench, a benchmark designed to evaluate Online Spatio-Temporal understanding from the perspective of an agent.<n>Built on an efficient data collection pipeline, OST-Bench consists of 1.4k scenes and 10k question-answer pairs collected from ScanNet, Matterport3D, and ARKitScenes.<n>We find that both complex-based spatial reasoning demands and long-term memory retrieval requirements significantly drop model performance along two separate axes.
  arXiv  Detail & Related papers  (2025-07-10T17:56:07Z)
• Exploiting Temporal State Space Sharing for Video Semantic Segmentation [53.8810901249897]
  Video semantic segmentation (VSS) plays a vital role in understanding the temporal evolution of scenes.<n>Traditional methods often segment videos frame-by-frame or in a short temporal window, leading to limited temporal context, redundant computations, and heavy memory requirements.<n>We introduce a Temporal Video State Space Sharing architecture to leverage Mamba state space models for temporal feature sharing.<n>Our model features a selective gating mechanism that efficiently propagates relevant information across video frames, eliminating the need for a memory-heavy feature pool.
  arXiv  Detail & Related papers  (2025-03-26T01:47:42Z)
• CAST: Cross-Attention in Space and Time for Video Action Recognition [8.785207228156098]
  We propose a novel two-stream architecture called Cross-Attention in Space and Time (CAST) CAST achieves a balanced spatial-temporal understanding of videos using only balanced input. Our proposed mechanism enables spatial and temporal expert models to exchange information and make synergistic predictions.
  arXiv  Detail & Related papers  (2023-11-30T18:58:51Z)
• You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos [56.676761067861236]
  Given an untrimmed video, temporal sentence grounding aims to locate a target moment semantically according to a sentence query. Previous respectable works have made decent success, but they only focus on high-level visual features extracted from decoded frames. We propose a new setting, compressed-domain TSG, which directly utilizes compressed videos rather than fully-decompressed frames as the visual input.
  arXiv  Detail & Related papers  (2023-03-14T12:53:27Z)
• Are We Ready for Vision-Centric Driving Streaming Perception? The ASAP Benchmark [23.872360763782037]
  ASAP is the first benchmark to evaluate the online performance of vision-centric perception in autonomous driving. We propose an annotation-extending pipeline to generate high-frame-rate labels for the 12Hz raw images. In the ASAP benchmark, comprehensive experiment results reveal that the model rank alters under different constraints.
  arXiv  Detail & Related papers  (2022-12-17T16:32:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.