Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologies

• URL: http://arxiv.org/abs/2312.11713v2
• Date: Wed, 24 Apr 2024 21:57:57 GMT
• Title: Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologies
• Authors: Jared Strader, Nathan Hughes, William Chen, Alberto Speranzon, Luca Carlone,
• Abstract summary: This paper proposes an approach to build 3D scene graphs in arbitrary indoor and outdoor environments. The hierarchy of concepts that describe an outdoor environment is more complex than for indoors. The lack of training data prevents the straightforward application of learning-based tools used in indoor settings.
• Score: 16.396336068724484
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper proposes an approach to build 3D scene graphs in arbitrary indoor and outdoor environments. Such extension is challenging; the hierarchy of concepts that describe an outdoor environment is more complex than for indoors, and manually defining such hierarchy is time-consuming and does not scale. Furthermore, the lack of training data prevents the straightforward application of learning-based tools used in indoor settings. To address these challenges, we propose two novel extensions. First, we develop methods to build a spatial ontology defining concepts and relations relevant for indoor and outdoor robot operation. In particular, we use a Large Language Model (LLM) to build such an ontology, thus largely reducing the amount of manual effort required. Second, we leverage the spatial ontology for 3D scene graph construction using Logic Tensor Networks (LTN) to add logical rules, or axioms (e.g., "a beach contains sand"), which provide additional supervisory signals at training time thus reducing the need for labelled data, providing better predictions, and even allowing predicting concepts unseen at training time. We test our approach in a variety of datasets, including indoor, rural, and coastal environments, and show that it leads to a significant increase in the quality of the 3D scene graph generation with sparsely annotated data.

Related papers

• Generalized Label-Efficient 3D Scene Parsing via Hierarchical Feature Aligned Pre-Training and Region-Aware Fine-tuning [55.517000360348725]
  This work presents a framework for dealing with 3D scene understanding when the labeled scenes are quite limited. To extract knowledge for novel categories from the pre-trained vision-language models, we propose a hierarchical feature-aligned pre-training and knowledge distillation strategy. Experiments with both indoor and outdoor scenes demonstrated the effectiveness of our approach in both data-efficient learning and open-world few-shot learning.
  arXiv  Detail & Related papers  (2023-12-01T15:47:04Z)
• ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning [125.90002884194838]
  ConceptGraphs is an open-vocabulary graph-structured representation for 3D scenes. It is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. We demonstrate the utility of this representation through a number of downstream planning tasks.
  arXiv  Detail & Related papers  (2023-09-28T17:53:38Z)
• 3D Annotation Of Arbitrary Objects In The Wild [0.0]
  We propose a data annotation pipeline based on SLAM, 3D reconstruction, and 3D-to-2D geometry. The pipeline allows creating 3D and 2D bounding boxes, along with per-pixel annotations of arbitrary objects. Our results showcase almost 90% Intersection-over-Union (IoU) agreement on both semantic segmentation and 2D bounding box detection.
  arXiv  Detail & Related papers  (2021-09-15T09:00:56Z)
• RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection [138.2892824662943]
  A promising solution is to make better use of the synthetic dataset, which consists of CAD object models, to boost the learning on real datasets. Recent work on 3D pre-training exhibits failure when transfer features learned on synthetic objects to other real-world applications. In this work, we put forward a new method called RandomRooms to accomplish this objective.
  arXiv  Detail & Related papers  (2021-08-17T17:56:12Z)
• Occlusion Guided Self-supervised Scene Flow Estimation on 3D Point Clouds [4.518012967046983]
  Understanding the flow in 3D space of sparsely sampled points between two consecutive time frames is the core stone of modern geometric-driven systems. This work presents a new self-supervised training method and an architecture for the 3D scene flow estimation under occlusions.
  arXiv  Detail & Related papers  (2021-04-10T09:55:19Z)
• Walk2Map: Extracting Floor Plans from Indoor Walk Trajectories [23.314557741879664]
  We present Walk2Map, a data-driven approach to generate floor plans from trajectories of a person walking inside the rooms. Thanks to advances in data-driven inertial odometry, such minimalistic input data can be acquired from the IMU readings of consumer-level smartphones. We train our networks using scanned 3D indoor models and apply them in a cascaded fashion on an indoor walk trajectory.
  arXiv  Detail & Related papers  (2021-02-27T16:29:09Z)
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding [107.02479689909164]
  In this work, we aim at facilitating research on 3D representation learning. We measure the effect of unsupervised pre-training on a large source set of 3D scenes.
  arXiv  Detail & Related papers  (2020-07-21T17:59:22Z)
• Mutual Information Maximization for Robust Plannable Representations [82.83676853746742]
  We present MIRO, an information theoretic representational learning algorithm for model-based reinforcement learning. We show that our approach is more robust than reconstruction objectives in the presence of distractors and cluttered scenes.
  arXiv  Detail & Related papers  (2020-05-16T21:58:47Z)
• 3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior [50.73148041205675]
  The goal of the Semantic Scene Completion (SSC) task is to simultaneously predict a completed 3D voxel representation of volumetric occupancy and semantic labels of objects in the scene from a single-view observation. We propose to devise a new geometry-based strategy to embed depth information with low-resolution voxel representation. Our proposed geometric embedding works better than the depth feature learning from habitual SSC frameworks.
  arXiv  Detail & Related papers  (2020-03-31T09:33:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.