Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphs

• URL: http://arxiv.org/abs/2606.31471v1
• Date: Tue, 30 Jun 2026 10:49:21 GMT
• Title: Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphs
• Abstract summary: Open-vocabulary 3D scene graph methods typically operate in two stages: first reconstruct, then enrich with vision-language models.<n>We propose an asynchronous architecture in which lightweight online mapping runs concurrently with heavyweight semantic refinement.<n>Our method matches or outperforms existing open-vocabulary 3D scene graph methods on semantic segmentation.
• Score: 10.390999817790425
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-vocabulary 3D scene graph methods typically operate in two stages: first reconstruct, then enrich with vision-language models, leaving the graph unqueryable during exploration. We argue that this sequential coupling is unnecessary and propose an asynchronous architecture in which lightweight online mapping runs concurrently with heavyweight semantic refinement. A probabilistic voxel-based backbone maintains stable object identities incrementally, while background VLM agents progressively enrich the graph. This framework resolves duplicate object tracks through semantic loop closure, attaches fine-grained visual attributes and derives spatial relations between objects. A multi-target frame scheduler amortizes VLM cost by selecting a small set of informative frames that jointly cover multiple targets. The resulting scene graph is queryable during exploration and grows in semantic richness over time. Our method matches or outperforms existing open-vocabulary 3D scene graph methods on semantic segmentation (ScanNet, Replica) and surpasses the prior state-of-the-art across three visual grounding benchmarks (Sr3D+, Nr3D, ScanRefer) by 15.3 to 18.8 A@0.25. Project page: https://denizbickici.github.io/thinkgraphs/

Related papers

• Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis [59.527737790821305]
  We propose a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding.<n>Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships.<n>Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability.
  arXiv  Detail & Related papers  (2026-07-06T17:29:49Z)
• SG2Loc: Sequential Visual Localization on 3D Scene Graphs [108.42363585468458]
  This paper introduces a novel, lightweight approach to sequential visual localization using 3D scene graphs.<n>By leveraging a compact scene graph and efficient semantic matching, our method significantly reduces storage while maintaining performance on real-world datasets.
  arXiv  Detail & Related papers  (2026-06-10T10:03:21Z)
• AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models [0.0]
  3D Visual Grounding (3DVG) is an essential capability for embodied AI, requiring agents to localize objects in 3D scenes based on natural language descriptions.<n>We present $bftextAgentGrounder$, a zero-shot 3D visual grounding framework that operates directly on colored point clouds without task-specific 3D training.
  arXiv  Detail & Related papers  (2026-05-25T14:29:04Z)
• SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching [2.4737219202679808]
  We propose SceneGrapher, a framework that reformulates 3D grounding as structured graph matching over a reconstructed 3D scene graph.<n>We demonstrate our framework through real-world deployment on a mobile robot, demonstrating robust spatial reasoning in long-horizon physical environments.
  arXiv  Detail & Related papers  (2026-05-20T22:30:51Z)
• ReLaGS: Relational Language Gaussian Splatting [20.136674901612334]
  We present a novel framework that constructs a hierarchical language-distilled Gaussian scene and its 3D semantic scene graph without scene-specific training.<n>On top of this hierarchy, we build an open-vocabulary 3D scene graph with Vision Language derived annotations and Graph Neural Network-based relational reasoning.<n>Our approach enables efficient and scalable open-vocabulary 3D reasoning by jointly modeling hierarchical semantics and inter/intra-object relationships.
  arXiv  Detail & Related papers  (2026-03-18T11:18:23Z)
• KeySG: Hierarchical Keyframe-Based 3D Scene Graphs [1.5134439544218246]
  KeySG represents 3D scenes as a hierarchical graph consisting of floors, rooms, objects, and functional elements.<n>We leverage VLM to extract scene information, alleviating the need to explicitly model relationship edges between objects.<n>Our approach can process complex and ambiguous queries while mitigating the scalability issues associated with large scene graphs.
  arXiv  Detail & Related papers  (2025-10-01T15:53:27Z)
• Open-Vocabulary Octree-Graph for 3D Scene Understanding [54.11828083068082]
  Octree-Graph is a novel scene representation for open-vocabulary 3D scene understanding. An adaptive-octree structure is developed that stores semantics and depicts the occupancy of an object adjustably according to its shape.
  arXiv  Detail & Related papers  (2024-11-25T10:14:10Z)
• Incremental 3D Semantic Scene Graph Prediction from RGB Sequences [86.77318031029404]
  We propose a real-time framework that incrementally builds a consistent 3D semantic scene graph of a scene given an RGB image sequence. Our method consists of a novel incremental entity estimation pipeline and a scene graph prediction network. The proposed network estimates 3D semantic scene graphs with iterative message passing using multi-view and geometric features extracted from the scene entities.
  arXiv  Detail & Related papers  (2023-05-04T11:32:16Z)
• SGAligner : 3D Scene Alignment with Scene Graphs [84.01002998166145]
  Building 3D scene graphs has emerged as a topic in scene representation for several embodied AI applications. We focus on the fundamental problem of aligning pairs of 3D scene graphs whose overlap can range from zero to partial. We propose SGAligner, the first method for aligning pairs of 3D scene graphs that is robust to in-the-wild scenarios.
  arXiv  Detail & Related papers  (2023-04-28T14:39:22Z)
• Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions [94.17683799712397]
  We focus on scene graphs, a data structure that organizes the entities of a scene in a graph. We propose a learned method that regresses a scene graph from the point cloud of a scene. We show the application of our method in a domain-agnostic retrieval task, where graphs serve as an intermediate representation for 3D-3D and 2D-3D matching.
  arXiv  Detail & Related papers  (2020-04-08T12:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.