RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots

• URL: http://arxiv.org/abs/2605.18197v1
• Date: Mon, 18 May 2026 10:37:39 GMT
• Title: RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots
• Abstract summary: Current approaches to 3D scene graph generation rely on dedicated depth sensors, such as LiDAR or RGB-D cameras, for metric 3D reconstruction.<n>This paper presents a fully visual framework for the active, incremental construction of 3D scene graphs from RGB input only.
• Score: 11.977612002830725
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current approaches to 3D scene graph generation rely on dedicated depth sensors, such as LiDAR or RGB-D cameras, for metric 3D reconstruction. This limits deployment to specialized robotic platforms and excludes settings where only RGB cameras are available, such as fixed external infrastructure. Existing pipelines also typically operate on passively collected observation trajectories, rather than selecting viewpoints based on the partially built scene representation, and therefore fail to effectively exploit the semantic and spatial information encoded within the graph during exploration. This paper presents a fully visual framework for the active, incremental construction of 3D scene graphs from RGB input only, addressing both limitations. The proposed approach unifies perception and planning around a shared structured representation that captures object semantics, 3D geometry, relational context, and information from multiple viewpoints. Because the framework is hardware-agnostic and relies only on RGB observations, it can incorporate inputs from both onboard robot cameras and fixed external cameras within the same representation. Experiments on the Replica dataset show that the RGB-only pipeline achieves F1-score parity with baselines using ground-truth depth. Active exploration experiments on ReplicaCAD further show that semantic-driven viewpoint selection detects more than twice as many objects as a geometric frontier-based baseline under the same exploration budget. Finally, the external-camera setting demonstrates that complementary RGB views can effectively bootstrap the scene graph and improve contextual understanding at no additional exploration cost.

Related papers

• StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation [4.871965587776022]
  We introduce StereoPatch, a patch-aligned RGB-depth representation that binds registered metric geometry directly to the RGB patches used for action prediction.<n>On a shared 2-D patch grid, asymmetric cross-attention incorporates depth information into the corresponding RGB features before action decoding.<n>Across six real-robot tasks, StereoPatch achieves higher closed-loop success than appearance-only, geometry-only, raw RGB-D, and late-fusion baselines.
  arXiv  Detail & Related papers  (2026-09-14T13:01:25Z)
• Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning [51.799524981291235]
  End-to-end manipulation policies show promise for generalizable and dexterous robotic manipulation.<n>They inherit two key limitations from 2D foundation models.<n>We present a series of contributions to address these issues.
  arXiv  Detail & Related papers  (2026-06-01T14:01:11Z)
• Fixed External Cameras as Common Prior Maps for Active 3D Scene Graph Generation [11.977612002830725]
  We present an RGB-only framework for active, incremental 3D scene graph generation that seamlessly fuses observations from both onboard robot cameras and fixed external cameras within a single hardware-agnostic pipeline.<n> Experiments demonstrate that bootstrapping the scene graph with even a single external camera increases initial object recall by up to +79%, and that the richer context of the prior significantly improves the efficiency of subsequent active exploration.
  arXiv  Detail & Related papers  (2026-05-18T10:26:07Z)
• Point2Graph: An End-to-end Point Cloud-based 3D Open-Vocabulary Scene Graph for Robot Navigation [7.048930431483034]
  Point2Graph is a novel end-to-end point cloud-based 3D open-vocabulary scene graph generation framework. For the room layer, we leverage the advantage of merging the geometry-based border detection algorithm with the learning-based region detection to segment rooms. In addition, we create an end-to-end pipeline for the object layer to detect and classify 3D objects based solely on 3D point cloud data.
  arXiv  Detail & Related papers  (2024-09-16T15:01:28Z)
• Anyview: Generalizable Indoor 3D Object Detection with Variable Frames [60.48134767838629]
  We present a novel 3D detection framework named AnyView for our practical applications.<n>Our method achieves both great generalizability and high detection accuracy with a simple and clean architecture.
  arXiv  Detail & Related papers  (2023-10-09T02:15:45Z)
• Artifacts Mapping: Multi-Modal Semantic Mapping for Object Detection and 3D Localization [13.473742114288616]
  We propose a framework that can autonomously detect and localize objects in a known environment. The framework consists of three key elements: understanding the environment through RGB data, estimating depth through multi-modal sensor fusion, and managing artifacts. Experiments show that the proposed framework can accurately detect 98% of the objects in the real sample environment, without post-processing.
  arXiv  Detail & Related papers  (2023-07-03T15:51:39Z)
• One-Shot Neural Fields for 3D Object Understanding [112.32255680399399]
  We present a unified and compact scene representation for robotics. Each object in the scene is depicted by a latent code capturing geometry and appearance. This representation can be decoded for various tasks such as novel view rendering, 3D reconstruction, and stable grasp prediction.
  arXiv  Detail & Related papers  (2022-10-21T17:33:14Z)
• Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision [51.385731364529306]
  This paper focuses on perceiving and navigating 3D environments using echoes and RGB image. In particular, we perform depth estimation by fusing RGB image with echoes, received from multiple orientations. We show that the echoes provide holistic and in-expensive information about the 3D structures complementing the RGB image.
  arXiv  Detail & Related papers  (2022-07-03T22:31:47Z)
• RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video [76.86512780916827]
  We present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline.
  arXiv  Detail & Related papers  (2021-06-22T12:53:56Z)
• Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images [69.5662419067878]
  Grounding referring expressions in RGBD image has been an emerging field. We present a novel task of 3D visual grounding in single-view RGBD image where the referred objects are often only partially scanned due to occlusion. Our approach first fuses the language and the visual features at the bottom level to generate a heatmap that localizes the relevant regions in the RGBD image. Then our approach conducts an adaptive feature learning based on the heatmap and performs the object-level matching with another visio-linguistic fusion to finally ground the referred object.
  arXiv  Detail & Related papers  (2021-03-14T11:18:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.