Estimating Generic 3D Room Structures from 2D Annotations

• URL: http://arxiv.org/abs/2306.09077v2
• Date: Thu, 21 Dec 2023 17:07:20 GMT
• Title: Estimating Generic 3D Room Structures from 2D Annotations
• Authors: Denys Rozumnyi, Stefan Popov, Kevis-Kokitsi Maninis, Matthias Nie{\ss}ner, Vittorio Ferrari
• Abstract summary: We propose a novel method to produce generic 3D room layouts just from 2D segmentation masks. Based on these 2D annotations, we automatically reconstruct 3D plane equations for the structural elements and their spatial extent in the scene. We release 2246 3D room layouts on the RealEstate10k dataset, containing YouTube videos.
• Score: 36.2713028459562
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Indoor rooms are among the most common use cases in 3D scene understanding. Current state-of-the-art methods for this task are driven by large annotated datasets. Room layouts are especially important, consisting of structural elements in 3D, such as wall, floor, and ceiling. However, they are difficult to annotate, especially on pure RGB video. We propose a novel method to produce generic 3D room layouts just from 2D segmentation masks, which are easy to annotate for humans. Based on these 2D annotations, we automatically reconstruct 3D plane equations for the structural elements and their spatial extent in the scene, and connect adjacent elements at the appropriate contact edges. We annotate and publicly release 2246 3D room layouts on the RealEstate10k dataset, containing YouTube videos. We demonstrate the high quality of these 3D layouts annotations with extensive experiments.

Related papers

• Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint [61.25279122171029]
  We present a framework that allows controllable and compositional 3D generation from text prompts. Our approach leverages 2D layouts as a blueprint to facilitate precise and plausible control over 3D generation.
  arXiv  Detail & Related papers  (2024-10-20T13:41:50Z)
• General Geometry-aware Weakly Supervised 3D Object Detection [62.26729317523975]
  A unified framework is developed for learning 3D object detectors from RGB images and associated 2D boxes. Experiments on KITTI and SUN-RGBD datasets demonstrate that our method yields surprisingly high-quality 3D bounding boxes with only 2D annotation.
  arXiv  Detail & Related papers  (2024-07-18T17:52:08Z)
• ControlRoom3D: Room Generation using Semantic Proxy Rooms [48.93419701713694]
  We present ControlRoom3D, a novel method to generate high-quality room meshes. Our approach is a user-defined 3D semantic proxy room that outlines a rough room layout. When rendered to 2D, this 3D representation provides valuable geometric and semantic information to control powerful 2D models.
  arXiv  Detail & Related papers  (2023-12-08T17:55:44Z)
• SceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation with Fine-Grained Geometry [92.24144643757963]
  3D indoor scenes are widely used in computer graphics, with applications ranging from interior design to gaming to virtual and augmented reality. High-quality 3D indoor scenes are highly demanded while it requires expertise and is time-consuming to design high-quality 3D indoor scenes manually. We propose SCENEHGN, a hierarchical graph network for 3D indoor scenes that takes into account the full hierarchy from the room level to the object level, then finally to the object part level. For the first time, our method is able to directly generate plausible 3D room content, including furniture objects with fine-grained geometry, and
  arXiv  Detail & Related papers  (2023-02-16T15:31:59Z)
• Towards 3D Object Detection with 2D Supervision [13.444432119639822]
  We introduce a hybrid training framework, enabling us to learn a visual 3D object detector with massive 2D labels. We propose a temporal 2D transformation to bridge the 3D predictions with temporal 2D labels. Experiments conducted on the nuScenes dataset show strong results (nearly 90% of its fully-supervised performance) with only 25% 3D annotations.
  arXiv  Detail & Related papers  (2022-11-15T16:40:11Z)
• Roominoes: Generating Novel 3D Floor Plans From Existing 3D Rooms [22.188206636953794]
  We propose the task of generating novel 3D floor plans from existing 3D rooms. One uses available 2D floor plans to guide selection and deformation of 3D rooms; the other learns to retrieve a set of compatible 3D rooms and combine them into novel layouts.
  arXiv  Detail & Related papers  (2021-12-10T16:17:01Z)
• Bidirectional Projection Network for Cross Dimension Scene Understanding [69.29443390126805]
  We present a emphbidirectional projection network (BPNet) for joint 2D and 3D reasoning in an end-to-end manner. Via the emphBPM, complementary 2D and 3D information can interact with each other in multiple architectural levels. Our emphBPNet achieves top performance on the ScanNetV2 benchmark for both 2D and 3D semantic segmentation.
  arXiv  Detail & Related papers  (2021-03-26T08:31:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.