Segment Any Architectural Facades (SAAF):An automatic segmentation model for building facades, walls and windows based on multimodal semantics guidance

• URL: http://arxiv.org/abs/2506.09071v2
• Date: Sat, 02 Aug 2025 12:21:14 GMT
• Title: Segment Any Architectural Facades (SAAF):An automatic segmentation model for building facades, walls and windows based on multimodal semantics guidance
• Authors: Peilin Li, Jun Yin, Jing Zhong, Ran Luo, Pengyu Zeng, Miao Zhang,
• Abstract summary: This study proposes an automatic segmentation model for building facade walls and windows based on multimodal semantic guidance.<n>We developed an end-to-end training framework that enables the model to autonomously learn the mapping relationship from text descriptions to image segmentation.<n>Our model has made certain progress in improving the accuracy and generalization ability of the wall and window segmentation task.
• Score: 17.461797749810327
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the context of the digital development of architecture, the automatic segmentation of walls and windows is a key step in improving the efficiency of building information models and computer-aided design. This study proposes an automatic segmentation model for building facade walls and windows based on multimodal semantic guidance, called Segment Any Architectural Facades (SAAF). First, SAAF has a multimodal semantic collaborative feature extraction mechanism. By combining natural language processing technology, it can fuse the semantic information in text descriptions with image features, enhancing the semantic understanding of building facade components. Second, we developed an end-to-end training framework that enables the model to autonomously learn the mapping relationship from text descriptions to image segmentation, reducing the influence of manual intervention on the segmentation results and improving the automation and robustness of the model. Finally, we conducted extensive experiments on multiple facade datasets. The segmentation results of SAAF outperformed existing methods in the mIoU metric, indicating that the SAAF model can maintain high-precision segmentation ability when faced with diverse datasets. Our model has made certain progress in improving the accuracy and generalization ability of the wall and window segmentation task. It is expected to provide a reference for the development of architectural computer vision technology and also explore new ideas and technical paths for the application of multimodal learning in the architectural field.

Related papers

• A Survey of Model Architectures in Information Retrieval [64.75808744228067]
  We focus on two key aspects: backbone models for feature extraction and end-to-end system architectures for relevance estimation.<n>We trace the development from traditional term-based methods to modern neural approaches, particularly highlighting the impact of transformer-based models and subsequent large language models (LLMs)<n>We conclude by discussing emerging challenges and future directions, including architectural optimizations for performance and scalability, handling of multimodal, multilingual data, and adaptation to novel application domains beyond traditional search paradigms.
  arXiv  Detail & Related papers  (2025-02-20T18:42:58Z)
• Interpretable deformable image registration: A geometric deep learning perspective [9.13809412085203]
  We present a theoretical foundation for designing an interpretable registration framework.<n>We formulate an end-to-end process that refines transformations in a coarse-to-fine fashion.<n>We conclude by showing significant improvement in performance metrics over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-12-17T19:47:10Z)
• Ensemble architecture in polyp segmentation [0.0]
  This study explores the architecture of semantic segmentation and evaluated models that excel in polyp segmentation. We present an integrated framework that harnesses the advantages of different models to attain an optimal outcome.
  arXiv  Detail & Related papers  (2024-08-14T02:57:38Z)
• Interfacing Foundation Models' Embeddings [131.0352288172788]
  We present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval.
  arXiv  Detail & Related papers  (2023-12-12T18:58:02Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations [70.41385310930846]
  We present an end-to-end framework Structure-CLIP to enhance multi-modal structured representations. We use scene graphs to guide the construction of semantic negative examples, which results in an increased emphasis on learning structured representations. A Knowledge-Enhance (KEE) is proposed to leverage SGK as input to further enhance structured representations.
  arXiv  Detail & Related papers  (2023-05-06T03:57:05Z)
• The Multiscale Surface Vision Transformer [10.833580445244094]
  We introduce the Multiscale Surface Vision Transformer (MS-SiT) as a backbone architecture for surface deep learning. Results demonstrate that the MS-SiT outperforms existing surface deep learning methods for neonatal phenotyping prediction tasks.
  arXiv  Detail & Related papers  (2023-03-21T15:00:17Z)
• Dynamically Grown Generative Adversarial Networks [111.43128389995341]
  We propose a method to dynamically grow a GAN during training, optimizing the network architecture and its parameters together with automation. The method embeds architecture search techniques as an interleaving step with gradient-based training to periodically seek the optimal architecture-growing strategy for the generator and discriminator.
  arXiv  Detail & Related papers  (2021-06-16T01:25:51Z)
• Edge-assisted Democratized Learning Towards Federated Analytics [67.44078999945722]
  We show the hierarchical learning structure of the proposed edge-assisted democratized learning mechanism, namely Edge-DemLearn. We also validate Edge-DemLearn as a flexible model training mechanism to build a distributed control and aggregation methodology in regions.
  arXiv  Detail & Related papers  (2020-12-01T11:46:03Z)
• End-to-End Trainable Deep Active Contour Models for Automated Image Segmentation: Delineating Buildings in Aerial Imagery [12.442780294349049]
  Trainable Deep Active Contours (TDACs) is an automatic image segmentation framework that unites Convolutional Networks (CNNs) and Active Contour Models (ACMs) TDAC yields fast, accurate, and fully automatic simultaneous delineation of arbitrarily many buildings in the image.
  arXiv  Detail & Related papers  (2020-07-22T21:27:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.