Vitruvio: 3D Building Meshes via Single Perspective Sketches

• URL: http://arxiv.org/abs/2210.13634v2
• Date: Tue, 11 Apr 2023 16:52:01 GMT
• Title: Vitruvio: 3D Building Meshes via Single Perspective Sketches
• Authors: Alberto Tono and Heyaojing Huang and Ashwin Agrawal and Martin Fischer
• Abstract summary: We introduce the first deep learning method focused only on buildings that aim to convert a single sketch to a 3D building mesh: Vitruvio. First, it accelerates the inference process by more than 26% (from 0.5s to 0.37s) Second, it increases the reconstruction accuracy (measured by the Chamfer Distance) by 18%.
• Score: 0.8001739956625484
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Today's architectural engineering and construction (AEC) software require a learning curve to generate a three-dimension building representation. This limits the ability to quickly validate the volumetric implications of an initial design idea communicated via a single sketch. Allowing designers to translate a single sketch to a 3D building will enable owners to instantly visualize 3D project information without the cognitive load required. If previous state-of-the-art (SOTA) data-driven methods for single view reconstruction (SVR) showed outstanding results in the reconstruction process from a single image or sketch, they lacked specific applications, analysis, and experiments in the AEC. Therefore, this research addresses this gap, introducing the first deep learning method focused only on buildings that aim to convert a single sketch to a 3D building mesh: Vitruvio. Vitruvio adapts Occupancy Network for SVR tasks on a specific building dataset (Manhattan 1K). This adaptation brings two main improvements. First, it accelerates the inference process by more than 26% (from 0.5s to 0.37s). Second, it increases the reconstruction accuracy (measured by the Chamfer Distance) by 18%. During this adaptation in the AEC domain, we evaluate the effect of the building orientation in the learning procedure since it constitutes an important design factor. While aligning all the buildings to a canonical pose improved the overall quantitative metrics, it did not capture fine-grain details in more complex building shapes (as shown in our qualitative analysis). Finally, Vitruvio outputs a 3D-printable building mesh with arbitrary topology and genus from a single perspective sketch, providing a step forward to allow owners and designers to communicate 3D information via a 2D, effective, intuitive, and universal communication medium: the sketch.

Related papers

• Exploiting Semantic Scene Reconstruction for Estimating Building Envelope Characteristics [6.382787013075262]
  We propose BuildNet3D, a novel framework to estimate geometric building characteristics from 2D image inputs. Our framework is evaluated on a range of complex building structures, demonstrating high accuracy and generalizability in estimating window-to-wall ratio and building footprint.
  arXiv  Detail & Related papers  (2024-10-29T13:29:01Z)
• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)
• RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation [50.35403070279804]
  3D occupancy prediction is an emerging task that aims to estimate the occupancy states and semantics of 3D scenes using multi-view images. We propose RadOcc, a Rendering assisted distillation paradigm for 3D Occupancy prediction.
  arXiv  Detail & Related papers  (2023-12-19T03:39:56Z)
• Semi-supervised Learning from Street-View Images and OpenStreetMap for Automatic Building Height Estimation [59.6553058160943]
  We propose a semi-supervised learning (SSL) method of automatically estimating building height from Mapillary SVI and OpenStreetMap data. The proposed method leads to a clear performance boosting in estimating building heights with a Mean Absolute Error (MAE) around 2.1 meters. The preliminary result is promising and motivates our future work in scaling up the proposed method based on low-cost VGI data.
  arXiv  Detail & Related papers  (2023-07-05T18:16:30Z)
• Automatic Reconstruction of Semantic 3D Models from 2D Floor Plans [1.8581514902689347]
  We present a pipeline for reconstruction of vectorized 3D models from scanned 2D plans. The method presented state-of-the-art results in the public dataset CubiCasa5k.
  arXiv  Detail & Related papers  (2023-06-02T16:06:42Z)
• 3D-LatentMapper: View Agnostic Single-View Reconstruction of 3D Shapes [0.0]
  We propose a novel framework that leverages the intermediate latent spaces of Vision Transformer (ViT) and a joint image-text representational model, CLIP, for fast and efficient Single View Reconstruction (SVR) We use the ShapeNetV2 dataset and perform extensive experiments with comparisons to SOTA methods to demonstrate our method's effectiveness.
  arXiv  Detail & Related papers  (2022-12-05T11:45:26Z)
• Structure-Aware 3D VR Sketch to 3D Shape Retrieval [113.20120789493217]
  We focus on the challenge caused by inherent inaccuracies in 3D VR sketches. We propose to use a triplet loss with an adaptive margin value driven by a "fitting gap" We introduce a dataset of 202 VR sketches for 202 3D shapes drawn from memory rather than from observation.
  arXiv  Detail & Related papers  (2022-09-19T14:29:26Z)
• Single-view 3D Mesh Reconstruction for Seen and Unseen Categories [69.29406107513621]
  Single-view 3D Mesh Reconstruction is a fundamental computer vision task that aims at recovering 3D shapes from single-view RGB images. This paper tackles Single-view 3D Mesh Reconstruction, to study the model generalization on unseen categories. We propose an end-to-end two-stage network, GenMesh, to break the category boundaries in reconstruction.
  arXiv  Detail & Related papers  (2022-08-04T14:13:35Z)
• Translational Symmetry-Aware Facade Parsing for 3D Building Reconstruction [11.263458202880038]
  In this paper, we present a novel translational symmetry-based approach to improving the deep neural networks. We propose a novel scheme to fuse anchor-free detection in a single stage network, which enables the efficient training and better convergence. We employ an off-the-shelf rendering engine like Blender to reconstruct the realistic high-quality 3D models using procedural modeling.
  arXiv  Detail & Related papers  (2021-06-02T03:10:51Z)
• Building-GAN: Graph-Conditioned Architectural Volumetric Design Generation [10.024367148266721]
  This paper focuses on volumetric design generation conditioned on an input program graph. Instead of outputting dense 3D voxels, we propose a new 3D representation named voxel graph that is both compact and expressive for building geometries. Our generator is a cross-modal graph neural network that uses a pointer mechanism to connect the input program graph and the output voxel graph, and the whole pipeline is trained using the adversarial framework.
  arXiv  Detail & Related papers  (2021-04-27T16:49:34Z)
• 3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior [50.73148041205675]
  The goal of the Semantic Scene Completion (SSC) task is to simultaneously predict a completed 3D voxel representation of volumetric occupancy and semantic labels of objects in the scene from a single-view observation. We propose to devise a new geometry-based strategy to embed depth information with low-resolution voxel representation. Our proposed geometric embedding works better than the depth feature learning from habitual SSC frameworks.
  arXiv  Detail & Related papers  (2020-03-31T09:33:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.