Habitat-Matterport 3D Semantics Dataset

• URL: http://arxiv.org/abs/2210.05633v3
• Date: Thu, 12 Oct 2023 18:40:37 GMT
• Title: Habitat-Matterport 3D Semantics Dataset
• Abstract summary: HM3DSEM is the largest dataset of 3D real-world spaces with densely annotated semantics currently available to the academic community. It consists of 142,646 object instance annotations across 216 3D spaces and 3,100 rooms within those spaces.
• Score: 72.14295757568439
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present the Habitat-Matterport 3D Semantics (HM3DSEM) dataset. HM3DSEM is the largest dataset of 3D real-world spaces with densely annotated semantics that is currently available to the academic community. It consists of 142,646 object instance annotations across 216 3D spaces and 3,100 rooms within those spaces. The scale, quality, and diversity of object annotations far exceed those of prior datasets. A key difference setting apart HM3DSEM from other datasets is the use of texture information to annotate pixel-accurate object boundaries. We demonstrate the effectiveness of HM3DSEM dataset for the Object Goal Navigation task using different methods. Policies trained using HM3DSEM perform outperform those trained on prior datasets. Introduction of HM3DSEM in the Habitat ObjectNav Challenge lead to an increase in participation from 400 submissions in 2021 to 1022 submissions in 2022.

Related papers

• Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence [78.1406635199656]
  Holi-Spatial is the first fully automated, large-scale, spatially-aware multimodal dataset constructed from raw video inputs without human intervention.<n>Holi-Spatial-4M is the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs.
  arXiv  Detail & Related papers  (2026-03-08T14:49:20Z)
• SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes [105.8644620467576]
  We introduce Stextscurprise3D, a novel dataset designed to evaluate language-guided spatial reasoning segmentation in complex 3D scenes.<n>Stextscurprise3D consists of more than 200k vision language pairs across 900+ detailed indoor scenes from ScanNet++ v2.<n>The dataset contains 89k+ human-annotated spatial queries deliberately crafted without object name.
  arXiv  Detail & Related papers  (2025-07-10T14:01:24Z)
• From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes [16.38713257618971]
  Anywhere3D-Bench is a holistic 3D visual grounding benchmark consisting of 2,632 referring expression-3D bounding box pairs.<n>We assess a range of state-of-the-art 3D visual grounding methods alongside large language models.
  arXiv  Detail & Related papers  (2025-06-05T11:28:02Z)
• SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Models [9.568997654206823]
  SORT3D is an approach that utilizes rich object attributes from 2D data and merges as-based spatial reasoning toolbox with the ability of large language models. We show that SORT3D achieves state-of-the-art performance on complex view-dependent grounding tasks on two benchmarks. We also implement the pipeline to run real-time on an autonomous vehicle and demonstrate that our approach can be used for object-goal navigation on previously unseen real-world environments.
  arXiv  Detail & Related papers  (2025-04-25T20:24:11Z)
• HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigation [39.54854283833085]
  We present the Habitat-Matterport 3D Open Vocabulary Object Goal Navigation dataset (HM3D-OVON) HM3D-OVON incorporates over 15k annotated instances of household objects across 379 distinct categories. We find that HM3D-OVON can be used to train an open-vocabulary ObjectNav agent that achieves both higher performance and is more robust to localization and actuation noise than the state-of-the-art ObjectNav approach.
  arXiv  Detail & Related papers  (2024-09-22T02:12:29Z)
• Task-oriented Sequential Grounding in 3D Scenes [35.90034571439091]
  We propose a new task: Task-oriented Sequential Grounding in 3D scenes. Agents must follow detailed step-by-step instructions to complete daily activities by locating a sequence of target objects in indoor scenes. To facilitate this task, we introduce SG3D, a large-scale dataset containing 22,346 tasks with 112,236 steps across 4,895 real-world 3D scenes.
  arXiv  Detail & Related papers  (2024-08-07T18:30:18Z)
• MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations [55.022519020409405]
  This paper builds the first largest ever multi-modal 3D scene dataset and benchmark with hierarchical grounded language annotations, MMScan. The resulting multi-modal 3D dataset encompasses 1.4M meta-annotated captions on 109k objects and 7.7k regions as well as over 3.04M diverse samples for 3D visual grounding and question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T17:59:30Z)
• Tracking Objects with 3D Representation from Videos [57.641129788552675]
  We propose a new 2D Multiple Object Tracking paradigm, called P3DTrack. With 3D object representation learning from Pseudo 3D object labels in monocular videos, we propose a new 2D MOT paradigm, called P3DTrack.
  arXiv  Detail & Related papers  (2023-06-08T17:58:45Z)
• Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI [55.475530452838946]
  HM3D contains 112.5k m2 of space, which is 1.4 - 3.7x larger than other building-scale datasets such as MP3D and Gibson. Images rendered from HM3D have 20 - 85% higher visual fidelity w.r.t. counterpart images captured with real cameras. HM3D-trained PointNav agents achieve 100% performance on Gibson-test dataset.
  arXiv  Detail & Related papers  (2021-09-16T22:01:24Z)
• D3D-HOI: Dynamic 3D Human-Object Interactions from Videos [49.38319295373466]
  We introduce D3D-HOI: a dataset of monocular videos with ground truth annotations of 3D object pose, shape and part motion during human-object interactions. Our dataset consists of several common articulated objects captured from diverse real-world scenes and camera viewpoints. We leverage the estimated 3D human pose for more accurate inference of the object spatial layout and dynamics.
  arXiv  Detail & Related papers  (2021-08-19T00:49:01Z)
• LayoutMP3D: Layout Annotation of Matterport3D [59.11106101006007]
  We consider the Matterport3D dataset with their originally provided depth map ground truths and further release our annotations for layout ground truths from a subset of Matterport3D. Our dataset provides both the layout and depth information, which enables the opportunity to explore the environment by integrating both cues.
  arXiv  Detail & Related papers  (2020-03-30T14:40:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.