Open-Vocabulary Octree-Graph for 3D Scene Understanding

• URL: http://arxiv.org/abs/2411.16253v1
• Date: Mon, 25 Nov 2024 10:14:10 GMT
• Title: Open-Vocabulary Octree-Graph for 3D Scene Understanding
• Abstract summary: Octree-Graph is a novel scene representation for open-vocabulary 3D scene understanding. An adaptive-octree structure is developed that stores semantics and depicts the occupancy of an object adjustably according to its shape.
• Score: 54.11828083068082
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-vocabulary 3D scene understanding is indispensable for embodied agents. Recent works leverage pretrained vision-language models (VLMs) for object segmentation and project them to point clouds to build 3D maps. Despite progress, a point cloud is a set of unordered coordinates that requires substantial storage space and does not directly convey occupancy information or spatial relation, making existing methods inefficient for downstream tasks, e.g., path planning and complex text-based object retrieval. To address these issues, we propose Octree-Graph, a novel scene representation for open-vocabulary 3D scene understanding. Specifically, a Chronological Group-wise Segment Merging (CGSM) strategy and an Instance Feature Aggregation (IFA) algorithm are first designed to get 3D instances and corresponding semantic features. Subsequently, an adaptive-octree structure is developed that stores semantics and depicts the occupancy of an object adjustably according to its shape. Finally, the Octree-Graph is constructed where each adaptive-octree acts as a graph node, and edges describe the spatial relations among nodes. Extensive experiments on various tasks are conducted on several widely-used datasets, demonstrating the versatility and effectiveness of our method.

Related papers

• Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis [59.527737790821305]
  We propose a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding.<n>Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships.<n>Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability.
  arXiv  Detail & Related papers  (2026-07-06T17:29:49Z)
• Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spaces [116.57196064763924]
  We extend the benchmark coverage by introducing dense tabletop objects and explicit multi-level functional relationships.<n>This work introduces critical challenges involving small-scale, dense, and similar instances.<n>We propose an open-vocabulary pipeline based on 2D visual grounding and 3D graph optimization.
  arXiv  Detail & Related papers  (2026-05-15T09:14:50Z)
• ReLaGS: Relational Language Gaussian Splatting [20.136674901612334]
  We present a novel framework that constructs a hierarchical language-distilled Gaussian scene and its 3D semantic scene graph without scene-specific training.<n>On top of this hierarchy, we build an open-vocabulary 3D scene graph with Vision Language derived annotations and Graph Neural Network-based relational reasoning.<n>Our approach enables efficient and scalable open-vocabulary 3D reasoning by jointly modeling hierarchical semantics and inter/intra-object relationships.
  arXiv  Detail & Related papers  (2026-03-18T11:18:23Z)
• KeySG: Hierarchical Keyframe-Based 3D Scene Graphs [1.5134439544218246]
  KeySG represents 3D scenes as a hierarchical graph consisting of floors, rooms, objects, and functional elements.<n>We leverage VLM to extract scene information, alleviating the need to explicitly model relationship edges between objects.<n>Our approach can process complex and ambiguous queries while mitigating the scalability issues associated with large scene graphs.
  arXiv  Detail & Related papers  (2025-10-01T15:53:27Z)
• 3D Part Segmentation via Geometric Aggregation of 2D Visual Features [57.20161517451834]
  Supervised 3D part segmentation models are tailored for a fixed set of objects and parts, limiting their transferability to open-set, real-world scenarios. Recent works have explored vision-language models (VLMs) as a promising alternative, using multi-view rendering and textual prompting to identify object parts. To address these limitations, we propose COPS, a COmprehensive model for Parts that blends semantics extracted from visual concepts and 3D geometry to effectively identify object parts.
  arXiv  Detail & Related papers  (2024-12-05T15:27:58Z)
• Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph [0.3926357402982764]
  Locating objects described in natural language presents a significant challenge for autonomous agents.<n>We propose a modular approach called BBQ that constructs 3D scene graph representation with metric and semantic spatial edges.<n>We show that BBQ takes a leading place in open-vocabulary 3D semantic segmentation compared to other zero-shot methods.
  arXiv  Detail & Related papers  (2024-06-11T09:57:04Z)
• SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Tasks [14.548198408544032]
  We treat 3D scene graph alignment as a partial graph-matching problem and propose to solve it with a graph neural network. We reuse the geometric features learned by a point cloud registration method and associate the clustered point-level geometric features with the node-level semantic feature. We propose a point-matching rescoring method, that uses the node-wise alignment of the 3D scene graph to reweight the matching candidates from a pre-trained point cloud registration method.
  arXiv  Detail & Related papers  (2024-03-28T15:01:58Z)
• LISNeRF Mapping: LiDAR-based Implicit Mapping via Semantic Neural Fields for Large-Scale 3D Scenes [2.822816116516042]
  Large-scale semantic mapping is crucial for outdoor autonomous agents to fulfill high-level tasks such as planning and navigation. This paper proposes a novel method for large-scale 3D semantic reconstruction through implicit representations from posed LiDAR measurements alone.
  arXiv  Detail & Related papers  (2023-11-04T03:55:38Z)
• ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning [125.90002884194838]
  ConceptGraphs is an open-vocabulary graph-structured representation for 3D scenes. It is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. We demonstrate the utility of this representation through a number of downstream planning tasks.
  arXiv  Detail & Related papers  (2023-09-28T17:53:38Z)
• Incremental 3D Semantic Scene Graph Prediction from RGB Sequences [86.77318031029404]
  We propose a real-time framework that incrementally builds a consistent 3D semantic scene graph of a scene given an RGB image sequence. Our method consists of a novel incremental entity estimation pipeline and a scene graph prediction network. The proposed network estimates 3D semantic scene graphs with iterative message passing using multi-view and geometric features extracted from the scene entities.
  arXiv  Detail & Related papers  (2023-05-04T11:32:16Z)
• SGAligner : 3D Scene Alignment with Scene Graphs [84.01002998166145]
  Building 3D scene graphs has emerged as a topic in scene representation for several embodied AI applications. We focus on the fundamental problem of aligning pairs of 3D scene graphs whose overlap can range from zero to partial. We propose SGAligner, the first method for aligning pairs of 3D scene graphs that is robust to in-the-wild scenarios.
  arXiv  Detail & Related papers  (2023-04-28T14:39:22Z)
• Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions [94.17683799712397]
  We focus on scene graphs, a data structure that organizes the entities of a scene in a graph. We propose a learned method that regresses a scene graph from the point cloud of a scene. We show the application of our method in a domain-agnostic retrieval task, where graphs serve as an intermediate representation for 3D-3D and 2D-3D matching.
  arXiv  Detail & Related papers  (2020-04-08T12:25:25Z)
• 3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior [50.73148041205675]
  The goal of the Semantic Scene Completion (SSC) task is to simultaneously predict a completed 3D voxel representation of volumetric occupancy and semantic labels of objects in the scene from a single-view observation. We propose to devise a new geometry-based strategy to embed depth information with low-resolution voxel representation. Our proposed geometric embedding works better than the depth feature learning from habitual SSC frameworks.
  arXiv  Detail & Related papers  (2020-03-31T09:33:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.