Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding

• URL: http://arxiv.org/abs/2608.21136v1
• Date: Fri, 21 Aug 2026 14:13:02 GMT
• Title: Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding
• Abstract summary: Stream3Dv2 is a training-free framework designed for robust streaming 3D perception.<n>We introduce a comprehensive geometric-semantic fusion mechanism that resolves geometric noise and semantic ambiguity.<n>Experiments on public datasets demonstrate that Stream3Dv2 consistently outperforms existing baselines in foundational open-vocabulary streaming 3D segmentation and detection.
• Score: 7.823301846592561
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, open-vocabulary zero-shot 3D scene understanding using vision foundation models has emerged as a promising alternative to data-intensive supervised methods. However, deploying these models in real-world scenarios is severely hindered by their inability to efficiently handle streaming RGB-D inputs and their inherent vulnerability to noise 2D segmentation masks. To address these critical limitations, we propose Stream3Dv2, a novel training-free framework designed for robust streaming 3D perception. Stream3Dv2 processes sequential data through an original nested local-to-historical architecture, capturing multi-view consistency while circumventing the high computational overhead so as to support timely responses. At its core, we introduce a comprehensive geometric-semantic fusion mechanism that resolves geometric noise and semantic ambiguity by explicitly utilizing semantic guidance and formulating 3D segmentation as solving point-and-set merging and partitioning problems. Furthermore, we present an innovative manifold-distance-based point cloud refinement strategy. This approach leverages local manifold graphs for point-to-manifold optimization that mitigates the boundary delineation failures caused by Euclidean-distance metrics, and employs geometric bounding boxes to dynamically activate and update historical instances for achieving rapid manifold-to-manifold refinement. Extensive experiments on public datasets demonstrate that Stream3Dv2 consistently outperforms existing baselines in foundational open-vocabulary streaming 3D segmentation and detection. Finally, we show that integrating our framework with an LLM-based agent enables advanced language-driven 3D scene understanding, underscoring its potential for open-world embodied intelligence. Code will be updated at https://github.com/SubmissionsIn/Stream3D.

Related papers

• Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spaces [116.57196064763924]
  We extend the benchmark coverage by introducing dense tabletop objects and explicit multi-level functional relationships.<n>This work introduces critical challenges involving small-scale, dense, and similar instances.<n>We propose an open-vocabulary pipeline based on 2D visual grounding and 3D graph optimization.
  arXiv  Detail & Related papers  (2026-05-15T09:14:50Z)
• LESV: Language Embedded Sparse Voxel Fusion for Open-Vocabulary 3D Scene Understanding [9.377694035678948]
  We present a novel framework that leverages Sparse Voxel Rasterization (SVRaster) as a structured, disjoint geometry representation.<n>This enables a deterministic, confidence-aware feature registration process and suppresses the semantic bleeding artifact common in 3DGS.<n>Our approach achieves state-of-the-art performance on Open Vocabulary 3D Object Retrieval and Point Cloud Understanding benchmarks.
  arXiv  Detail & Related papers  (2026-04-01T20:48:06Z)
• 3D-IDE: 3D Implicit Depth Emergent [45.72771473431863]
  3D-Implicit Depth Emergence is a method that reframes 3D perception as an emergent property derived from geometric self-vision.<n>Our method enables 3D perception to emerge implicitly, disentangling features in dense regions.<n>Our approach achieves a 55% reduction in inference latency while maintaining strong performance across diverse downstream tasks.
  arXiv  Detail & Related papers  (2026-03-28T00:54:19Z)
• Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation [34.44214123004662]
  We propose VLM3D, a framework for differentiable semantic and spatial critics.<n>Our core contribution is a dual-language critic signal derived from the VLM's Yes or No log-odds.<n>VLM3D establishes a principled and general path to inject the VLM's rich, language-grounded understanding of both semantics and space into diverse 3D generative pipelines.
  arXiv  Detail & Related papers  (2025-11-18T09:05:26Z)
• EA3D: Online Open-World 3D Object Extraction from Streaming Videos [55.48835711373918]
  We present ExtractAnything3D (EA3D), a unified online framework for open-world 3D object extraction.<n>Given a streaming video, EA3D dynamically interprets each frame using vision-language and 2D vision foundation encoders to extract object-level knowledge.<n>A recurrent joint optimization module directs the model's attention to regions of interest, simultaneously enhancing both geometric reconstruction and semantic understanding.
  arXiv  Detail & Related papers  (2025-10-29T03:56:41Z)
• Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding [58.38294408121273]
  We propose Cross-modal and Uncertainty-aware Agglomeration for Open-vocabulary 3D Scene Understanding dubbed CUA-O3D.<n>Our method addresses two key challenges: (1) incorporating semantic priors from VLMs alongside the geometric knowledge of spatially-aware vision foundation models, and (2) using a novel deterministic uncertainty estimation to capture model-specific uncertainties.
  arXiv  Detail & Related papers  (2025-03-20T20:58:48Z)
• ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction [62.599588577671796]
  We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality.
  arXiv  Detail & Related papers  (2023-11-29T20:30:18Z)
• Flattening-Net: Deep Regular 2D Representation for 3D Point Cloud Analysis [66.49788145564004]
  We present an unsupervised deep neural architecture called Flattening-Net to represent irregular 3D point clouds of arbitrary geometry and topology. Our methods perform favorably against the current state-of-the-art competitors.
  arXiv  Detail & Related papers  (2022-12-17T15:05:25Z)
• OPA-3D: Occlusion-Aware Pixel-Wise Aggregation for Monocular 3D Object Detection [51.153003057515754]
  OPA-3D is a single-stage, end-to-end, Occlusion-Aware Pixel-Wise Aggregation network. It jointly estimates dense scene depth with depth-bounding box residuals and object bounding boxes. It outperforms state-of-the-art methods on the main Car category.
  arXiv  Detail & Related papers  (2022-11-02T14:19:13Z)
• Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perception [122.53774221136193]
  State-of-the-art methods for driving-scene LiDAR-based perception often project the point clouds to 2D space and then process them via 2D convolution. A natural remedy is to utilize the 3D voxelization and 3D convolution network. We propose a new framework for the outdoor LiDAR segmentation, where cylindrical partition and asymmetrical 3D convolution networks are designed to explore the 3D geometric pattern.
  arXiv  Detail & Related papers  (2021-09-12T06:25:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.