Scan2LoD3: Reconstructing semantic 3D building models at LoD3 using ray
casting and Bayesian networks
- URL: http://arxiv.org/abs/2305.06314v1
- Date: Wed, 10 May 2023 17:01:18 GMT
- Title: Scan2LoD3: Reconstructing semantic 3D building models at LoD3 using ray
casting and Bayesian networks
- Authors: Olaf Wysocki, Yan Xia, Magdalena Wysocki, Eleonora Grilli, Ludwig
Hoegner, Daniel Cremers, Uwe Stilla
- Abstract summary: Reconstructing semantic 3D building models at the level of detail (LoD) 3 is a long-standing challenge.
We present a novel method, called Scan2LoD3, that accurately reconstructs semantic LoD3 building models.
We believe our method can foster the development of probability-driven semantic 3D reconstruction at LoD3.
- Score: 40.7734793392562
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Reconstructing semantic 3D building models at the level of detail (LoD) 3 is
a long-standing challenge. Unlike mesh-based models, they require watertight
geometry and object-wise semantics at the fa\c{c}ade level. The principal
challenge of such demanding semantic 3D reconstruction is reliable
fa\c{c}ade-level semantic segmentation of 3D input data. We present a novel
method, called Scan2LoD3, that accurately reconstructs semantic LoD3 building
models by improving fa\c{c}ade-level semantic 3D segmentation. To this end, we
leverage laser physics and 3D building model priors to probabilistically
identify model conflicts. These probabilistic physical conflicts propose
locations of model openings: Their final semantics and shapes are inferred in a
Bayesian network fusing multimodal probabilistic maps of conflicts, 3D point
clouds, and 2D images. To fulfill demanding LoD3 requirements, we use the
estimated shapes to cut openings in 3D building priors and fit semantic 3D
objects from a library of fa\c{c}ade objects. Extensive experiments on the TUM
city campus datasets demonstrate the superior performance of the proposed
Scan2LoD3 over the state-of-the-art methods in fa\c{c}ade-level detection,
semantic segmentation, and LoD3 building model reconstruction. We believe our
method can foster the development of probability-driven semantic 3D
reconstruction at LoD3 since not only the high-definition reconstruction but
also reconstruction confidence becomes pivotal for various applications such as
autonomous driving and urban simulations.
Related papers
- DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction [67.13370009386635]
We introduce the Dual Point Maps (DualPM), where a pair of point maps is extracted from the same image, one associating pixels to their 3D locations on the object, and the other to a canonical version of the object at rest pose.
We show that 3D reconstruction and 3D pose estimation reduce to the prediction of the DualPMs.
arXiv Detail & Related papers (2024-12-05T18:59:48Z) - Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulation [30.744137117668643]
Lift3D is a framework that enhances 2D foundation models with implicit and explicit 3D robotic representations to construct a robust 3D manipulation policy.
In experiments, Lift3D consistently outperforms previous state-of-the-art methods across several simulation benchmarks and real-world scenarios.
arXiv Detail & Related papers (2024-11-27T18:59:52Z) - Combining visibility analysis and deep learning for refinement of
semantic 3D building models by conflict classification [3.2662392450935416]
We propose a method of combining visibility analysis and neural networks for enriching 3D models with window and door features.
In the method, occupancy voxels are fused with classified point clouds, which provides semantics to voxels.
The semantic voxels and conflicts are combined in a Bayesian network to classify and delineate faccade openings, which are reconstructed using a 3D model library.
arXiv Detail & Related papers (2023-03-10T16:01:30Z) - MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices [78.20154723650333]
High-quality 3D ground-truth shapes are critical for 3D object reconstruction evaluation.
We introduce a novel multi-view RGBD dataset captured using a mobile device.
We obtain precise 3D ground-truth shape without relying on high-end 3D scanners.
arXiv Detail & Related papers (2023-03-03T14:02:50Z) - 3D Shape Reconstruction from 2D Images with Disentangled Attribute Flow [61.62796058294777]
Reconstructing 3D shape from a single 2D image is a challenging task.
Most of the previous methods still struggle to extract semantic attributes for 3D reconstruction task.
We propose 3DAttriFlow to disentangle and extract semantic attributes through different semantic levels in the input images.
arXiv Detail & Related papers (2022-03-29T02:03:31Z) - DensePose 3D: Lifting Canonical Surface Maps of Articulated Objects to
the Third Dimension [71.71234436165255]
We contribute DensePose 3D, a method that can learn such reconstructions in a weakly supervised fashion from 2D image annotations only.
Because it does not require 3D scans, DensePose 3D can be used for learning a wide range of articulated categories such as different animal species.
We show significant improvements compared to state-of-the-art non-rigid structure-from-motion baselines on both synthetic and real data on categories of humans and animals.
arXiv Detail & Related papers (2021-08-31T18:33:55Z) - A Convolutional Architecture for 3D Model Embedding [1.3858051019755282]
We propose a deep learning architecture to handle 3D models as an input.
We show that the embedding representation conveys semantic information that helps to deal with the similarity assessment of 3D objects.
arXiv Detail & Related papers (2021-03-05T15:46:47Z) - An Effective Loss Function for Generating 3D Models from Single 2D Image
without Rendering [0.0]
Differentiable rendering is a very successful technique that applies to a Single-View 3D Reconstruction.
Currents use losses based on pixels between a rendered image of some 3D reconstructed object and ground-truth images from given matched viewpoints to optimise parameters of the 3D shape.
We propose a novel effective loss function that evaluates how well the projections of reconstructed 3D point clouds cover the ground truth object's silhouette.
arXiv Detail & Related papers (2021-03-05T00:02:18Z) - Implicit Functions in Feature Space for 3D Shape Reconstruction and
Completion [53.885984328273686]
Implicit Feature Networks (IF-Nets) deliver continuous outputs, can handle multiple topologies, and complete shapes for missing or sparse input data.
IF-Nets clearly outperform prior work in 3D object reconstruction in ShapeNet, and obtain significantly more accurate 3D human reconstructions.
arXiv Detail & Related papers (2020-03-03T11:14:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.