Synthesizing 3D Abstractions by Inverting Procedural Buildings with Transformers

• URL: http://arxiv.org/abs/2501.17044v2
• Date: Wed, 29 Jan 2025 11:06:57 GMT
• Title: Synthesizing 3D Abstractions by Inverting Procedural Buildings with Transformers
• Authors: Maximilian Dax, Jordi Berbel, Jan Stria, Leonidas Guibas, Urs Bergmann,
• Abstract summary: We generate abstractions of buildings by learning to invert procedural models.<n>Our approach achieves good reconstruction accuracy in terms of geometry and structure, as well as structurally consistent inpainting.
• Score: 2.199128905898291
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We generate abstractions of buildings, reflecting the essential aspects of their geometry and structure, by learning to invert procedural models. We first build a dataset of abstract procedural building models paired with simulated point clouds and then learn the inverse mapping through a transformer. Given a point cloud, the trained transformer then infers the corresponding abstracted building in terms of a programmatic language description. This approach leverages expressive procedural models developed for gaming and animation, and thereby retains desirable properties such as efficient rendering of the inferred abstractions and strong priors for regularity and symmetry. Our approach achieves good reconstruction accuracy in terms of geometry and structure, as well as structurally consistent inpainting.

Related papers

• HORT: Monocular Hand-held Objects Reconstruction with Transformers [61.36376511119355]
  Reconstructing hand-held objects in 3D from monocular images is a significant challenge in computer vision. We propose a transformer-based model to efficiently reconstruct dense 3D point clouds of hand-held objects. Our method achieves state-of-the-art accuracy with much faster inference speed, while generalizing well to in-the-wild images.
  arXiv  Detail & Related papers  (2025-03-27T09:45:09Z)
• ArcPro: Architectural Programs for Structured 3D Abstraction of Sparse Points [21.052495894521872]
  ArcPro is a learning framework built on architectural programs to recover structured 3D abstractions from point clouds. We train an encoder-decoder on the points-program pairs to establish a mapping from unstructured point clouds to architectural programs. Inference by our method is highly efficient and produces plausible and faithful 3D abstractions.
  arXiv  Detail & Related papers  (2025-03-04T16:10:42Z)
• Learning Fine-to-Coarse Cuboid Shape Abstraction [7.152103069753289]
  abstraction of 3D objects with simple geometric primitives like cuboids allows to infer structural information from complex geometry. We introduce a novel fine-to-coarse unsupervised learning approach to abstract collections of 3D shapes. Our results confirm an improvement over previous cuboid-based shape abstraction techniques.
  arXiv  Detail & Related papers  (2025-02-03T22:06:26Z)
• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• DeFormer: Integrating Transformers with Deformable Models for 3D Shape Abstraction from a Single Image [31.154786931081087]
  We propose a novel bi-channel Transformer architecture, integrated with parameterized deformable models, to simultaneously estimate the global and local deformations of primitives. DeFormer achieves better reconstruction accuracy over the state-of-the-art, and visualizes with consistent semantic correspondences for improved interpretability.
  arXiv  Detail & Related papers  (2023-09-22T02:46:43Z)
• ANISE: Assembly-based Neural Implicit Surface rEconstruction [12.745433575962842]
  We present ANISE, a method that reconstructs a 3Dshape from partial observations (images or sparse point clouds) The shape is formulated as an assembly of neural implicit functions, each representing a different part instance. We demonstrate that, when performing reconstruction by decoding part representations into implicit functions, our method achieves state-of-the-art part-aware reconstruction results from both images and sparse point clouds.
  arXiv  Detail & Related papers  (2022-05-27T00:01:40Z)
• Autoregressive 3D Shape Generation via Canonical Mapping [92.91282602339398]
  transformers have shown remarkable performances in a variety of generative tasks such as image, audio, and text generation. In this paper, we aim to further exploit the power of transformers and employ them for the task of 3D point cloud generation. Our model can be easily extended to multi-modal shape completion as an application for conditional shape generation.
  arXiv  Detail & Related papers  (2022-04-05T03:12:29Z)
• Automated LoD-2 Model Reconstruction from Very-HighResolution Satellite-derived Digital Surface Model and Orthophoto [1.2691047660244335]
  We propose a model-driven method that reconstructs LoD-2 building models following a "decomposition-optimization-fitting" paradigm. Our proposed method has addressed a few technical caveats over existing methods, resulting in practically high-quality results.
  arXiv  Detail & Related papers  (2021-09-08T19:03:09Z)
• Translational Symmetry-Aware Facade Parsing for 3D Building Reconstruction [11.263458202880038]
  In this paper, we present a novel translational symmetry-based approach to improving the deep neural networks. We propose a novel scheme to fuse anchor-free detection in a single stage network, which enables the efficient training and better convergence. We employ an off-the-shelf rendering engine like Blender to reconstruct the realistic high-quality 3D models using procedural modeling.
  arXiv  Detail & Related papers  (2021-06-02T03:10:51Z)
• Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks [118.20778308823779]
  We present a novel 3D primitive representation that defines primitives using an Invertible Neural Network (INN) Our model learns to parse 3D objects into semantically consistent part arrangements without any part-level supervision.
  arXiv  Detail & Related papers  (2021-03-18T17:59:31Z)
• Convolutional Occupancy Networks [88.48287716452002]
  We propose Convolutional Occupancy Networks, a more flexible implicit representation for detailed reconstruction of objects and 3D scenes. By combining convolutional encoders with implicit occupancy decoders, our model incorporates inductive biases, enabling structured reasoning in 3D space. We empirically find that our method enables the fine-grained implicit 3D reconstruction of single objects, scales to large indoor scenes, and generalizes well from synthetic to real data.
  arXiv  Detail & Related papers  (2020-03-10T10:17:07Z)
• Unsupervised Learning of Intrinsic Structural Representation Points [50.92621061405056]
  Learning structures of 3D shapes is a fundamental problem in the field of computer graphics and geometry processing. We present a simple yet interpretable unsupervised method for learning a new structural representation in the form of 3D structure points.
  arXiv  Detail & Related papers  (2020-03-03T17:40:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.