S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction

• URL: http://arxiv.org/abs/2603.14232v1
• Date: Sun, 15 Mar 2026 05:48:55 GMT
• Title: S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction
• Abstract summary: Streaming Semantic Gaussian Splatting (S2GS) is a strictly causal, incremental 3D Gaussian semantic field framework.<n>It does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames.<n>S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks.
• Score: 57.07346645250984
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing offline feed-forward methods for joint scene understanding and reconstruction on long image streams often repeatedly perform global computation over an ever-growing set of past observations, causing runtime and GPU memory to increase rapidly with sequence length and limiting scalability. We propose Streaming Semantic Gaussian Splatting (S2GS), a strictly causal, incremental 3D Gaussian semantic field framework: it does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames, enabling scalable online joint reconstruction and understanding. S2GS adopts a geometry-semantic decoupled dual-backbone design: the geometry branch performs causal modeling to drive incremental Gaussian updates, while the semantic branch leverages a 2D foundation vision model and a query-driven decoder to predict segmentation masks and identity embeddings, further stabilized by query-level contrastive alignment and lightweight online association with an instance memory. Experiments show that S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks, while significantly improving long-horizon scalability: it processes 1,000+ frames with much slower growth in runtime and GPU memory, whereas offline global-processing baselines typically run out of memory at around 80 frames under the same setting.

Related papers

• Rethinking Memory as Continuously Evolving Connectivity [63.58181883607843]
  Existing memory-augmented LLM agents treat memory as a static repository with pre-defined representations and fixed retrieval pipelines.<n>We propose FluxMem, a connectivity-evolving memory framework that models memory as a heterogeneous graph and progressively refines its topology through three stages.<n>Across three fundamentally distinct benchmarks including LoCoMo, Mind2Web, and GAIA, FluxMem achieves consistent state-of-the-art performance, demonstrating strong adaptation and generalization in complex agentic environments.
  arXiv  Detail & Related papers  (2026-05-27T17:35:34Z)
• Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval [86.05617542701644]
  Visual Geometry Grounded Transformer (VGGT) advances 3D reconstruction via scalable Transformer architecture.<n>StreamVGGT enables streaming with causal attention, yet its KV cache grows linearly with frames, causing memory overflow and quality degradation.<n>We present RetrieveVGGT, a training-free framework, which formulates context construction for VGGT as a retrieval problem.
  arXiv  Detail & Related papers  (2026-05-10T16:41:54Z)
• Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding [55.7992006853979]
  SAVEMem is a framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.<n>We present SAVEMem, a training-free dual-stage framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.
  arXiv  Detail & Related papers  (2026-05-08T15:40:40Z)
• Ray-Aware Pointer Memory with Adaptive Updates for Streaming 3D Reconstruction [19.132257919770947]
  3D reconstruction from continuous image streams requires both accurate geometric aggregation and stable memory management.<n>We propose a ray-aware pointer memory for streaming 3D reconstruction that explicitly models both spatial location and viewing direction.<n>Our approach provides a principled framework for scalable and drift-resistant online 3D reconstruction from image streams.
  arXiv  Detail & Related papers  (2026-05-07T06:42:11Z)
• Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction [50.5449251266956]
  This paper addresses the task of large-scale 3D scene reconstruction from long video sequences.<n>Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints.<n>We propose a novel neural global context representation that efficiently compresses and retains long-range scene information.
  arXiv  Detail & Related papers  (2026-04-09T17:59:50Z)
• LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows [10.300202521638274]
  We introduce the Large Sparse Reconstruction Model to study how scaling transformer context windows impacts feed-forward 3D reconstruction.<n>We show that expanding the context window -- by substantially increasing the number of active object and image tokens -- remarkably narrows this gap and enables high-fidelity 3D object reconstruction and inverse rendering.
  arXiv  Detail & Related papers  (2026-04-06T21:21:12Z)
• LoGeR: Long-Context Geometric Reconstruction with Hybrid Memory [97.14005794889134]
  We present LoGeR, a novel architecture that scales dense 3D reconstruction to extremely long sequences without post-optimization.<n>LoGeR processes video streams in chunks, leveraging strong bidirectional priors for high-fidelity intra-chunk reasoning.<n>This memory architecture enables LoGeR to be trained on sequences of 128 frames, and generalize up to thousands of frames during inference.
  arXiv  Detail & Related papers  (2026-03-03T18:55:37Z)
• RELIC: Interactive Video World Model with Long-Horizon Memory [74.81433479334821]
  A truly interactive world model requires real-time long-horizon streaming, consistent spatial memory, and precise user control.<n>We present RELIC, a unified framework that tackles these three challenges altogether.<n>Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time.
  arXiv  Detail & Related papers  (2025-12-03T18:29:20Z)
• TGSFormer: Scalable Temporal Gaussian Splatting for Embodied Semantic Scene Completion [38.31128137855632]
  Embodied 3D Semantic Scene Completion infers dense geometry and semantics from continuous egocentric observations.<n>Recent depth-guided approach alleviates this issue but remains local, suffering from latency and memory overhead as scale increases.<n>We propose TGSFormer, a scalable Temporal Gaussian Splatting framework for embodied SSC.
  arXiv  Detail & Related papers  (2025-11-29T03:47:14Z)
• Local2Global query Alignment for Video Instance Segmentation [6.422775545814375]
  Video segmentation methods excel at handling long sequences and capturing gradual changes, making them ideal for real-world applications.<n>This paper introduces Local2Global, an online framework, for instance segmentation, exhibiting state-of-the-art performance with simple baseline and training purely in online fashion.<n>We propose the L2G-aligner, a novel lightweight transformer decoder, to facilitate an early alignment between local and global queries.
  arXiv  Detail & Related papers  (2025-07-27T04:04:01Z)
• StreamGS: Online Generalizable Gaussian Splatting Reconstruction for Unposed Image Streams [32.91936079359693]
  We propose StreamGS, an online generalizable 3DGS reconstruction method for unposed image streams.<n>StreamGS transforms image streams to 3D Gaussian streams by predicting and aggregating per-frame Gaussians.<n>Experiments on diverse datasets have demonstrated that StreamGS achieves quality on par with optimization-based approaches but does so 150 times faster.
  arXiv  Detail & Related papers  (2025-03-08T14:35:39Z)
• InstantSplat: Sparse-view Gaussian Splatting in Seconds [91.77050739918037]
  We introduce InstantSplat, a novel approach for addressing sparse-view 3D scene reconstruction at lightning-fast speed.<n>InstantSplat employs a self-supervised framework that optimize 3D scene representation and camera poses.<n>It achieves an acceleration of over 30x in reconstruction and improves visual quality (SSIM) from 0.3755 to 0.7624 compared to traditional SfM with 3D-GS.
  arXiv  Detail & Related papers  (2024-03-29T17:29:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.