Multi-View Depth Consistent Image Generation Using Generative AI Models: Application on Architectural Design of University Buildings

• URL: http://arxiv.org/abs/2503.03068v1
• Date: Wed, 05 Mar 2025 00:16:09 GMT
• Title: Multi-View Depth Consistent Image Generation Using Generative AI Models: Application on Architectural Design of University Buildings
• Authors: Xusheng Du, Ruihan Gui, Zhengyang Wang, Ye Zhang, Haoran Xie,
• Abstract summary: We propose a novel three-stage consistent image generation framework using generative AI models.<n>We employ ControlNet as the backbone and optimize it to accommodate multi-view inputs of architectural shoebox models.<n> Experimental results demonstrate that the proposed framework can generate multi-view architectural images with consistent style and structural coherence.
• Score: 20.569648863933285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the early stages of architectural design, shoebox models are typically used as a simplified representation of building structures but require extensive operations to transform them into detailed designs. Generative artificial intelligence (AI) provides a promising solution to automate this transformation, but ensuring multi-view consistency remains a significant challenge. To solve this issue, we propose a novel three-stage consistent image generation framework using generative AI models to generate architectural designs from shoebox model representations. The proposed method enhances state-of-the-art image generation diffusion models to generate multi-view consistent architectural images. We employ ControlNet as the backbone and optimize it to accommodate multi-view inputs of architectural shoebox models captured from predefined perspectives. To ensure stylistic and structural consistency across multi-view images, we propose an image space loss module that incorporates style loss, structural loss and angle alignment loss. We then use depth estimation method to extract depth maps from the generated multi-view images. Finally, we use the paired data of the architectural images and depth maps as inputs to improve the multi-view consistency via the depth-aware 3D attention module. Experimental results demonstrate that the proposed framework can generate multi-view architectural images with consistent style and structural coherence from shoebox model inputs.

Related papers

• CDI3D: Cross-guided Dense-view Interpolation for 3D Reconstruction [25.468907201804093]
  Large Reconstruction Models (LRMs) have shown great promise in leveraging multi-view images generated by 2D diffusion models to extract 3D content. However, 2D diffusion models often struggle to produce dense images with strong multi-view consistency. We present CDI3D, a feed-forward framework designed for efficient, high-quality image-to-3D generation with view.
  arXiv  Detail & Related papers  (2025-03-11T03:08:43Z)
• IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations [64.07859467542664]
  Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics.<n>Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs.<n>We introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations.
  arXiv  Detail & Related papers  (2024-12-16T18:52:56Z)
• From Flat to Spatial: Comparison of 4 methods constructing 3D, 2 and 1/2D Models from 2D Plans with neural networks [0.0]
  The conversion of single images into 2 and 1/2D and 3D meshes is a promising technology that enhances design visualization and efficiency. This paper evaluates four innovative methods: "One-2-3-45," " CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model," "Instant Mesh," and "Image-to-Mesh"
  arXiv  Detail & Related papers  (2024-07-29T13:01:20Z)
• GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement [51.97726804507328]
  We propose a novel approach for 3D mesh reconstruction from multi-view images. Our method takes inspiration from large reconstruction models that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images.
  arXiv  Detail & Related papers  (2024-06-09T05:19:24Z)
• Distance Weighted Trans Network for Image Completion [52.318730994423106]
  We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. CNNs are used to augment the local texture information of coarse priors. DWT blocks are used to recover certain coarse textures and coherent visual structures.
  arXiv  Detail & Related papers  (2023-10-11T12:46:11Z)
• StructuredMesh: 3D Structured Optimization of Fa\c{c}ade Components on Photogrammetric Mesh Models using Binary Integer Programming [17.985961236568663]
  We present StructuredMesh, a novel approach for reconstructing faccade structures conforming to the regularity of buildings within photogrammetric mesh models. Our method involves capturing multi-view color and depth images of the building model using a virtual camera. We then utilize the depth image to remap these boxes into 3D space, generating an initial faccade layout.
  arXiv  Detail & Related papers  (2023-06-07T06:40:54Z)
• CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training [112.96224800952724]
  We propose cascaded modulation GAN (CM-GAN) to generate plausible image structures when dealing with large holes in complex images. In each decoder block, global modulation is first applied to perform coarse semantic-aware synthesis structure, then spatial modulation is applied on the output of global modulation to further adjust the feature map in a spatially adaptive fashion. In addition, we design an object-aware training scheme to prevent the network from hallucinating new objects inside holes, fulfilling the needs of object removal tasks in real-world scenarios.
  arXiv  Detail & Related papers  (2022-03-22T16:13:27Z)
• Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE [74.29384873537587]
  We propose a two-stage model for diverse inpainting, where the first stage generates multiple coarse results each of which has a different structure, and the second stage refines each coarse result separately by augmenting texture. Experimental results on CelebA-HQ, Places2, and ImageNet datasets show that our method not only enhances the diversity of the inpainting solutions but also improves the visual quality of the generated multiple images.
  arXiv  Detail & Related papers  (2021-03-18T05:10:49Z)
• Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN [52.851990439671475]
  We propose a novel end-to-end GAN architecture that can generate high-resolution 3D images. We achieve this goal by using different configurations between training and inference. Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation.
  arXiv  Detail & Related papers  (2020-08-05T02:33:04Z)
• Generating Annotated High-Fidelity Images Containing Multiple Coherent Objects [10.783993190686132]
  We propose a multi-object generation framework that can synthesize images with multiple objects without explicitly requiring contextual information. We demonstrate how coherency and fidelity are preserved with our method through experiments on the Multi-MNIST and CLEVR datasets.
  arXiv  Detail & Related papers  (2020-06-22T11:33:55Z)
• Learning Deformable Image Registration from Optimization: Perspective, Modules, Bilevel Training and Beyond [62.730497582218284]
  We develop a new deep learning based framework to optimize a diffeomorphic model via multi-scale propagation. We conduct two groups of image registration experiments on 3D volume datasets including image-to-atlas registration on brain MRI data and image-to-image registration on liver CT data.
  arXiv  Detail & Related papers  (2020-04-30T03:23:45Z)
• Multimodal Deep Unfolding for Guided Image Super-Resolution [23.48305854574444]
  Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a high-resolution output. We propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture. Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information.
  arXiv  Detail & Related papers  (2020-01-21T14:41:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.