TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances

• URL: http://arxiv.org/abs/2412.05596v2
• Date: Mon, 24 Feb 2025 10:27:39 GMT
• Title: TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances
• Authors: Wenting Xu, Viorela Ila, Luping Zhou, Craig T. Jin,
• Abstract summary: We develop a model that learns to structure and vary functional affordance across a 3D hierarchical scene graph.<n>The varying functional affordance is designed to integrate with the varying spatial context of the graph.
• Score: 20.4157915852084
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The concept of function and affordance is a critical aspect of 3D scene understanding and supports task-oriented objectives. In this work, we develop a model that learns to structure and vary functional affordance across a 3D hierarchical scene graph representing the spatial organization of a scene. The varying functional affordance is designed to integrate with the varying spatial context of the graph. More specifically, we develop an algorithm that learns to construct a 3D hierarchical scene graph (3DHSG) that captures the spatial organization of the scene. Starting from segmented object point clouds and object semantic labels, we develop a 3DHSG with a top node that identifies the room label, child nodes that define local spatial regions inside the room with region-specific affordances, and grand-child nodes indicating object locations and object-specific affordances. To support this work, we create a custom 3DHSG dataset that provides ground truth data for local spatial regions with region-specific affordances and also object-specific affordances for each object. We employ a transformer-based model to learn the 3DHSG. We use a multi-task learning framework that learns both room classification and learns to define spatial regions within the room with region-specific affordances. Our work improves on the performance of state-of-the-art baseline models and shows one approach for applying transformer models to 3D scene understanding and the generation of 3DHSGs that capture the spatial organization of a room. The code and dataset are publicly available.

Related papers

• Gaussian Splatting Feature Fields for Privacy-Preserving Visual Localization [29.793562435104707]
  We propose a scene representation for visual localization that combines an explicit geometry model (3DGS) with an implicit feature field.<n>We use a 3D structure-informed clustering procedure to regularize the representation learning and seamlessly convert the features to segmentations.<n>The resulting privacy- and non-privacy-preserving localization pipelines, evaluated on multiple real-world datasets, show state-of-the-art performances.
  arXiv  Detail & Related papers  (2025-07-31T13:58:15Z)
• Open-Vocabulary Indoor Object Grounding with 3D Hierarchical Scene Graph [0.0]
  OVIGo-3DHSG represents an extensive indoor environment over a Hierarchical Scene Graph.<n>The hierarchical representation explicitly models spatial relations across floors, rooms, locations, and objects.<n>Our approach demonstrates efficient scene comprehension and robust object grounding compared to existing methods.
  arXiv  Detail & Related papers  (2025-07-16T10:47:12Z)
• SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes [105.8644620467576]
  We introduce Stextscurprise3D, a novel dataset designed to evaluate language-guided spatial reasoning segmentation in complex 3D scenes.<n>Stextscurprise3D consists of more than 200k vision language pairs across 900+ detailed indoor scenes from ScanNet++ v2.<n>The dataset contains 89k+ human-annotated spatial queries deliberately crafted without object name.
  arXiv  Detail & Related papers  (2025-07-10T14:01:24Z)
• Intelligent Spatial Perception by Building Hierarchical 3D Scene Graphs for Indoor Scenarios with the Help of LLMs [4.764379183672723]
  This paper introduces a novel system that harnesses the capabilities of Large Language Models (LLMs) to construct hierarchical 3D Scene Graphs for indoor scenarios. The proposed framework constructs 3DSGs consisting of a fundamental layer with rich metric-semantic information, an object layer featuring precise point-cloud representation of object nodes, and higher layers of room, floor, and building nodes. Thanks to the innovative application of LLMs, not only object nodes but also nodes of higher layers, e.g., room nodes, are annotated in an intelligent and accurate manner.
  arXiv  Detail & Related papers  (2025-03-19T10:40:28Z)
• Open-Vocabulary Octree-Graph for 3D Scene Understanding [54.11828083068082]
  Octree-Graph is a novel scene representation for open-vocabulary 3D scene understanding. An adaptive-octree structure is developed that stores semantics and depicts the occupancy of an object adjustably according to its shape.
  arXiv  Detail & Related papers  (2024-11-25T10:14:10Z)
• 3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding [58.924180772480504]
  3D visual grounding aims to localize the target object in a 3D point cloud by a free-form language description. We propose a relation-aware one-stage framework, named 3D Relative Position-aware Network (3-Net)
  arXiv  Detail & Related papers  (2023-07-25T09:33:25Z)
• Generating Visual Spatial Description via Holistic 3D Scene Understanding [88.99773815159345]
  Visual spatial description (VSD) aims to generate texts that describe the spatial relations of the given objects within images. With an external 3D scene extractor, we obtain the 3D objects and scene features for input images. We construct a target object-centered 3D spatial scene graph (Go3D-S2G), such that we model the spatial semantics of target objects within the holistic 3D scenes.
  arXiv  Detail & Related papers  (2023-05-19T15:53:56Z)
• SceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation with Fine-Grained Geometry [92.24144643757963]
  3D indoor scenes are widely used in computer graphics, with applications ranging from interior design to gaming to virtual and augmented reality. High-quality 3D indoor scenes are highly demanded while it requires expertise and is time-consuming to design high-quality 3D indoor scenes manually. We propose SCENEHGN, a hierarchical graph network for 3D indoor scenes that takes into account the full hierarchy from the room level to the object level, then finally to the object part level. For the first time, our method is able to directly generate plausible 3D room content, including furniture objects with fine-grained geometry, and
  arXiv  Detail & Related papers  (2023-02-16T15:31:59Z)
• Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search [48.655167907740136]
  We use a 3D scene graph representation to capture the hierarchical, semantic, and geometric aspects of this problem. We introduce Hierarchical Mechanical Search (HMS), a method that guides an agent's actions towards finding a target object specified with a natural language description. HMS is evaluated on a novel dataset of 500 3D scene graphs with dense placements of semantically related objects in storage locations.
  arXiv  Detail & Related papers  (2020-12-07T21:04:34Z)
• Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions [94.17683799712397]
  We focus on scene graphs, a data structure that organizes the entities of a scene in a graph. We propose a learned method that regresses a scene graph from the point cloud of a scene. We show the application of our method in a domain-agnostic retrieval task, where graphs serve as an intermediate representation for 3D-3D and 2D-3D matching.
  arXiv  Detail & Related papers  (2020-04-08T12:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.