FacaDiffy: Inpainting Unseen Facade Parts Using Diffusion Models

• URL: http://arxiv.org/abs/2502.14940v1
• Date: Thu, 20 Feb 2025 17:32:41 GMT
• Title: FacaDiffy: Inpainting Unseen Facade Parts Using Diffusion Models
• Abstract summary: We introduce FacaDiffy, a novel method for inpainting unseen facade parts by completing conflict maps with a personalized Stable Diffusion model.<n>Specifically, we first propose a deterministic ray analysis approach to derive 2D conflict maps from existing 3D building models and corresponding laser scanning point clouds.<n>To complement the scarcity of real-world training data, we also develop a scalable pipeline to produce synthetic conflict maps using random city model generators and annotated facade images.
• Score: 33.30545717892921
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High-detail semantic 3D building models are frequently utilized in robotics, geoinformatics, and computer vision. One key aspect of creating such models is employing 2D conflict maps that detect openings' locations in building facades. Yet, in reality, these maps are often incomplete due to obstacles encountered during laser scanning. To address this challenge, we introduce FacaDiffy, a novel method for inpainting unseen facade parts by completing conflict maps with a personalized Stable Diffusion model. Specifically, we first propose a deterministic ray analysis approach to derive 2D conflict maps from existing 3D building models and corresponding laser scanning point clouds. Furthermore, we facilitate the inpainting of unseen facade objects into these 2D conflict maps by leveraging the potential of personalizing a Stable Diffusion model. To complement the scarcity of real-world training data, we also develop a scalable pipeline to produce synthetic conflict maps using random city model generators and annotated facade images. Extensive experiments demonstrate that FacaDiffy achieves state-of-the-art performance in conflict map completion compared to various inpainting baselines and increases the detection rate by $22\%$ when applying the completed conflict maps for high-definition 3D semantic building reconstruction. The code is be publicly available in the corresponding GitHub repository: https://github.com/ThomasFroech/InpaintingofUnseenFacadeObjects

Related papers

• ObjFiller-3D: Consistent Multi-view 3D Inpainting via Video Diffusion Models [76.80262068405243]
  3D inpainting often relies on multi-view 2D image inpainting, where inherent inconsistencies result in blurred textures, spatial discontinuities, and distracting visual artifacts.<n>We propose Filler-3D, a novel method designed for the completion and editing of high-quality and consistent 3D objects.<n>We analyze the representation gap between 3D and videos, and propose an adaptation of a video inpainting model for 3D scene inpainting.
  arXiv  Detail & Related papers  (2025-08-25T17:59:40Z)
• CM2LoD3: Reconstructing LoD3 Building Models Using Semantic Conflict Maps [0.0]
  Detailed 3D building models are crucial for urban planning, digital twins, and disaster management applications.<n>While Level of Detail 1 (LoD)1 and LoD2 building models are widely available, they lack detailed facade elements essential for advanced urban analysis.<n>We present a novel method for reconstructing LoD3 building models leveraging Conflict Maps (CMs) obtained from ray-to-model-prior analysis.
  arXiv  Detail & Related papers  (2025-08-21T15:54:13Z)
• Learning to Align and Refine: A Foundation-to-Diffusion Framework for Occlusion-Robust Two-Hand Reconstruction [50.952228546326516]
  Two-hand reconstruction from monocular images faces persistent challenges due to complex and dynamic hand postures.<n>Existing approaches struggle with such alignment issues, often resulting in misalignment and penetration artifacts.<n>We propose a dual-stage Foundation-to-Diffusion framework that precisely align 2D prior guidance from vision foundation models.
  arXiv  Detail & Related papers  (2025-03-22T14:42:27Z)
• SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images [49.7344030427291]
  We study the problem of single-image 3D object reconstruction.<n>Recent works have diverged into two directions: regression-based modeling and generative modeling.<n>We present SPAR3D, a novel two-stage approach aiming to take the best of both directions.
  arXiv  Detail & Related papers  (2025-01-08T18:52:03Z)
• Dust to Tower: Coarse-to-Fine Photo-Realistic Scene Reconstruction from Sparse Uncalibrated Images [11.1039786318131]
  Dust to Tower (D2T) is an efficient framework to optimize 3DGS and image poses simultaneously from sparse and uncalibrated images.<n>Our key idea is to first construct a coarse model efficiently and subsequently refine it using warped and inpainted images at novel viewpoints.<n>Experiments and ablation studies demonstrate the validity of D2T and its design choices.
  arXiv  Detail & Related papers  (2024-12-27T08:19:34Z)
• DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction [67.13370009386635]
  We introduce the Dual Point Maps (DualPM), where a pair of point maps is extracted from the same image, one associating pixels to their 3D locations on the object, and the other to a canonical version of the object at rest pose.<n>We show that 3D reconstruction and 3D pose estimation reduce to the prediction of the DualPMs.
  arXiv  Detail & Related papers  (2024-12-05T18:59:48Z)
• Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion [77.34078223594686]
  We propose a novel architecture for direct 3D scene generation by introducing diffusion models into 3D sparse representations and combining them with neural rendering techniques. Specifically, our approach generates texture colors at the point level for a given geometry using a 3D diffusion model first, which is then transformed into a scene representation in a feed-forward manner. Experiments in two city-scale datasets show that our model demonstrates proficiency in generating photo-realistic street-view image sequences and cross-view urban scenes from satellite imagery.
  arXiv  Detail & Related papers  (2024-01-19T16:15:37Z)
• NeRFiller: Completing Scenes via Generative 3D Inpainting [113.18181179986172]
  We propose NeRFiller, an approach that completes missing portions of a 3D capture via generative 3D inpainting. In contrast to related works, we focus on completing scenes rather than deleting foreground objects.
  arXiv  Detail & Related papers  (2023-12-07T18:59:41Z)
• Iterative Superquadric Recomposition of 3D Objects from Multiple Views [77.53142165205283]
  We propose a framework, ISCO, to recompose an object using 3D superquadrics as semantic parts directly from 2D views. Our framework iteratively adds new superquadrics wherever the reconstruction error is high. It provides consistently more accurate 3D reconstructions, even from images in the wild.
  arXiv  Detail & Related papers  (2023-09-05T10:21:37Z)
• O$^2$-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model [28.372289119872764]
  Occlusion is a common issue in 3D reconstruction from RGB-D videos, often blocking the complete reconstruction of objects. We propose a novel framework, empowered by a 2D diffusion-based in-painting model, to reconstruct complete surfaces for the hidden parts of objects.
  arXiv  Detail & Related papers  (2023-08-18T14:38:31Z)
• Scan2LoD3: Reconstructing semantic 3D building models at LoD3 using ray casting and Bayesian networks [40.7734793392562]
  Reconstructing semantic 3D building models at the level of detail (LoD) 3 is a long-standing challenge. We present a novel method, called Scan2LoD3, that accurately reconstructs semantic LoD3 building models. We believe our method can foster the development of probability-driven semantic 3D reconstruction at LoD3.
  arXiv  Detail & Related papers  (2023-05-10T17:01:18Z)
• Combining visibility analysis and deep learning for refinement of semantic 3D building models by conflict classification [3.2662392450935416]
  We propose a method of combining visibility analysis and neural networks for enriching 3D models with window and door features. In the method, occupancy voxels are fused with classified point clouds, which provides semantics to voxels. The semantic voxels and conflicts are combined in a Bayesian network to classify and delineate faccade openings, which are reconstructed using a 3D model library.
  arXiv  Detail & Related papers  (2023-03-10T16:01:30Z)
• DensePose 3D: Lifting Canonical Surface Maps of Articulated Objects to the Third Dimension [71.71234436165255]
  We contribute DensePose 3D, a method that can learn such reconstructions in a weakly supervised fashion from 2D image annotations only. Because it does not require 3D scans, DensePose 3D can be used for learning a wide range of articulated categories such as different animal species. We show significant improvements compared to state-of-the-art non-rigid structure-from-motion baselines on both synthetic and real data on categories of humans and animals.
  arXiv  Detail & Related papers  (2021-08-31T18:33:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.