Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI

• URL: http://arxiv.org/abs/2109.08238v1
• Date: Thu, 16 Sep 2021 22:01:24 GMT
• Title: Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
• Abstract summary: HM3D contains 112.5k m2 of space, which is 1.4 - 3.7x larger than other building-scale datasets such as MP3D and Gibson. Images rendered from HM3D have 20 - 85% higher visual fidelity w.r.t. counterpart images captured with real cameras. HM3D-trained PointNav agents achieve 100% performance on Gibson-test dataset.
• Score: 55.475530452838946
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present the Habitat-Matterport 3D (HM3D) dataset. HM3D is a large-scale dataset of 1,000 building-scale 3D reconstructions from a diverse set of real-world locations. Each scene in the dataset consists of a textured 3D mesh reconstruction of interiors such as multi-floor residences, stores, and other private indoor spaces. HM3D surpasses existing datasets available for academic research in terms of physical scale, completeness of the reconstruction, and visual fidelity. HM3D contains 112.5k m^2 of navigable space, which is 1.4 - 3.7x larger than other building-scale datasets such as MP3D and Gibson. When compared to existing photorealistic 3D datasets such as Replica, MP3D, Gibson, and ScanNet, images rendered from HM3D have 20 - 85% higher visual fidelity w.r.t. counterpart images captured with real cameras, and HM3D meshes have 34 - 91% fewer artifacts due to incomplete surface reconstruction. The increased scale, fidelity, and diversity of HM3D directly impacts the performance of embodied AI agents trained using it. In fact, we find that HM3D is `pareto optimal' in the following sense -- agents trained to perform PointGoal navigation on HM3D achieve the highest performance regardless of whether they are evaluated on HM3D, Gibson, or MP3D. No similar claim can be made about training on other datasets. HM3D-trained PointNav agents achieve 100% performance on Gibson-test dataset, suggesting that it might be time to retire that episode dataset.

Related papers

• Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs [62.865932175477035]
  Metric 3D object detection is a core capability for embodied agents, yet most reliable systems lean on depth sensors.<n>We propose Map-Det3D, an online multi-view 3D object detection model that brings detection directly into a 3D space reconstructed from RGB.
  arXiv  Detail & Related papers  (2026-08-12T15:33:42Z)
• RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion [49.933001840775816]
  RaySt3R recasts 3D shape completion as a novel view synthesis problem.<n>We train a feedforward transformer to predict depth maps, object masks, and per-pixel confidence scores for query rays.<n>RaySt3R fuses these predictions across multiple query views to reconstruct complete 3D shapes.
  arXiv  Detail & Related papers  (2025-06-05T17:43:23Z)
• 3D-LLM: Injecting the 3D World into Large Language Models [60.43823088804661]
  Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. We propose to inject the 3D world into large language models and introduce a new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks.
  arXiv  Detail & Related papers  (2023-07-24T17:59:02Z)
• MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices [78.20154723650333]
  High-quality 3D ground-truth shapes are critical for 3D object reconstruction evaluation. We introduce a novel multi-view RGBD dataset captured using a mobile device. We obtain precise 3D ground-truth shape without relying on high-end 3D scanners.
  arXiv  Detail & Related papers  (2023-03-03T14:02:50Z)
• OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation [107.71752592196138]
  We propose OmniObject3D, a large vocabulary 3D object dataset with massive high-quality real-scanned 3D objects. It comprises 6,000 scanned objects in 190 daily categories, sharing common classes with popular 2D datasets. Each 3D object is captured with both 2D and 3D sensors, providing textured meshes, point clouds, multiview rendered images, and multiple real-captured videos.
  arXiv  Detail & Related papers  (2023-01-18T18:14:18Z)
• Habitat-Matterport 3D Semantics Dataset [72.14295757568439]
  HM3DSEM is the largest dataset of 3D real-world spaces with densely annotated semantics currently available to the academic community. It consists of 142,646 object instance annotations across 216 3D spaces and 3,100 rooms within those spaces.
  arXiv  Detail & Related papers  (2022-10-11T17:25:51Z)
• Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild [32.05421669957098]
  Large datasets and scalable solutions have led to unprecedented advances in 2D recognition. We revisit the task of 3D object detection by introducing a large benchmark, called Omni3D. We show that Cube R-CNN outperforms prior works on the larger Omni3D and existing benchmarks.
  arXiv  Detail & Related papers  (2022-07-21T17:56:22Z)
• HM3D-ABO: A Photo-realistic Dataset for Object-centric Multi-view 3D Reconstruction [37.29140654256627]
  We present a photo-realistic object-centric dataset HM3D-ABO. It is constructed by composing realistic indoor scene and realistic object. The dataset could also be useful for tasks such as camera pose estimation and novel-view synthesis.
  arXiv  Detail & Related papers  (2022-06-24T16:02:01Z)
• 3D-to-2D Distillation for Indoor Scene Parsing [78.36781565047656]
  We present a new approach that enables us to leverage 3D features extracted from large-scale 3D data repository to enhance 2D features extracted from RGB images. First, we distill 3D knowledge from a pretrained 3D network to supervise a 2D network to learn simulated 3D features from 2D features during the training. Second, we design a two-stage dimension normalization scheme to calibrate the 2D and 3D features for better integration. Third, we design a semantic-aware adversarial training model to extend our framework for training with unpaired 3D data.
  arXiv  Detail & Related papers  (2021-04-06T02:22:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.