Relationship-Aware Hierarchical 3D Scene Graph for Task Reasoning

• URL: http://arxiv.org/abs/2602.02456v1
• Date: Mon, 02 Feb 2026 18:47:02 GMT
• Title: Relationship-Aware Hierarchical 3D Scene Graph for Task Reasoning
• Abstract summary: We propose an enhanced hierarchical 3D scene graph that integrates open-vocabulary features across multiple abstraction levels and supports object-relational reasoning.<n>We introduce a task reasoning module that combines Large Language Models (LLM) and a Vision Language Model (VLM) to interpret the scene graph's semantic and relational information.<n>We validate our method by deploying it on a quadruped robot in multiple environments and tasks, highlighting its ability to reason about them.
• Score: 10.364779390403337
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Representing and understanding 3D environments in a structured manner is crucial for autonomous agents to navigate and reason about their surroundings. While traditional Simultaneous Localization and Mapping (SLAM) methods generate metric reconstructions and can be extended to metric-semantic mapping, they lack a higher level of abstraction and relational reasoning. To address this gap, 3D scene graphs have emerged as a powerful representation for capturing hierarchical structures and object relationships. In this work, we propose an enhanced hierarchical 3D scene graph that integrates open-vocabulary features across multiple abstraction levels and supports object-relational reasoning. Our approach leverages a Vision Language Model (VLM) to infer semantic relationships. Notably, we introduce a task reasoning module that combines Large Language Models (LLM) and a VLM to interpret the scene graph's semantic and relational information, enabling agents to reason about tasks and interact with their environment more intelligently. We validate our method by deploying it on a quadruped robot in multiple environments and tasks, highlighting its ability to reason about them.

Related papers

• Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis [59.527737790821305]
  We propose a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding.<n>Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships.<n>Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability.
  arXiv  Detail & Related papers  (2026-07-06T17:29:49Z)
• From Pixels to Concepts: Growing Rich 3D Semantic Scene Graph Forests utilizing Foundation Models [4.137761255401348]
  Hierarchical 3D scene graphs integrate geometric, semantic, and relational data within a unified spatial framework.<n>Current 3D scene graph approaches often restrict themselves to rigid structures of pre-determined relationship classes.<n>This paper explores the potential of foundation models to build forests of 3D scene graphs with open semantic relationships.
  arXiv  Detail & Related papers  (2026-06-22T13:26:17Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spaces [116.57196064763924]
  We extend the benchmark coverage by introducing dense tabletop objects and explicit multi-level functional relationships.<n>This work introduces critical challenges involving small-scale, dense, and similar instances.<n>We propose an open-vocabulary pipeline based on 2D visual grounding and 3D graph optimization.
  arXiv  Detail & Related papers  (2026-05-15T09:14:50Z)
• Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM [71.01178857331969]
  Chat-Scene++ is an MLLM framework that represents 3D scenes as context-rich object sequences.<n>By structuring scenes as sequences of objects with contextual semantics, Chat-Scene++ enables object-centric representation and interaction.<n>Chat-Scene++ achieves state-of-the-art performance on five major 3D vision-language benchmarks.
  arXiv  Detail & Related papers  (2026-03-29T04:16:04Z)
• ReLaGS: Relational Language Gaussian Splatting [20.136674901612334]
  We present a novel framework that constructs a hierarchical language-distilled Gaussian scene and its 3D semantic scene graph without scene-specific training.<n>On top of this hierarchy, we build an open-vocabulary 3D scene graph with Vision Language derived annotations and Graph Neural Network-based relational reasoning.<n>Our approach enables efficient and scalable open-vocabulary 3D reasoning by jointly modeling hierarchical semantics and inter/intra-object relationships.
  arXiv  Detail & Related papers  (2026-03-18T11:18:23Z)
• Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning [24.17324180628543]
  We propose a unified framework for Open-World 3D Scene Graph Generation with Retrieval-Augmented Reasoning.<n>Our method integrates Vision-Language Models (VLMs) with retrieval-based reasoning to support multimodal exploration and language-guided interaction.<n>We evaluate our method on 3DSSG and Replica benchmarks across four tasks-scene question answering, visual grounding, instance retrieval, and task planning-demonstrating robust generalization and superior performance in diverse environments.
  arXiv  Detail & Related papers  (2025-11-08T07:37:29Z)
• KeySG: Hierarchical Keyframe-Based 3D Scene Graphs [1.5134439544218246]
  KeySG represents 3D scenes as a hierarchical graph consisting of floors, rooms, objects, and functional elements.<n>We leverage VLM to extract scene information, alleviating the need to explicitly model relationship edges between objects.<n>Our approach can process complex and ambiguous queries while mitigating the scalability issues associated with large scene graphs.
  arXiv  Detail & Related papers  (2025-10-01T15:53:27Z)
• A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding [78.99798110890157]
  Open-vocabulary 3D visual grounding aims to localize target objects based on free-form language queries.<n>Existing language field methods struggle to accurately localize instances using spatial relations in language queries.<n>We propose SpatialReasoner, a novel neural representation-based framework with large language model (LLM)-driven spatial reasoning.
  arXiv  Detail & Related papers  (2025-07-09T10:20:38Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)
• DSM: Constructing a Diverse Semantic Map for 3D Visual Grounding [4.016097197373127]
  We introduce the Diverse Semantic Map (DSM) framework, a novel scene representation framework that enriches robust geometric models with a spectrum of VLM-derived semantics.<n>On the ScanRefer benchmark, DSM-Grounding achieves a state-of-the-art 59.06% overall accuracy of IoU@0.5, surpassing others by 10%.<n>In semantic segmentation, our DSM attains a 67.93% F-mIoU, outperforming all baselines, including privileged ones.
  arXiv  Detail & Related papers  (2025-04-11T07:18:42Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning [125.90002884194838]
  ConceptGraphs is an open-vocabulary graph-structured representation for 3D scenes. It is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. We demonstrate the utility of this representation through a number of downstream planning tasks.
  arXiv  Detail & Related papers  (2023-09-28T17:53:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.