Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery

• URL: http://arxiv.org/abs/2605.26381v1
• Date: Mon, 25 May 2026 23:01:05 GMT
• Title: Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery
• Abstract summary: We present a multi-modal classification framework that fuses satellite and street-level imagery through a Perceiver IO architecture.<n>We construct a large-scale dataset of 32,135 buildings spanning ten countries, pairing satellite images with up to eight street-level views per segment.<n>The Perceiver IO fusion model improves over all other fusion strategies and yields substantial per-class gains for attributes visible from street level.
• Score: 18.60580673640467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a multi-modal classification framework that fuses satellite and street-level imagery through a Perceiver IO architecture operating on spatial patch tokens from a shared DINOv2 backbone. The design naturally handles a variable number of street-level views per building without padding or fixed-size pooling, and jointly predicts multi-label roof element and roof material classes. We construct a large-scale dataset of 32,135 buildings (61,672 segments) spanning ten countries, pairing satellite images with up to eight street-level views per segment and evaluating four masking strategies for isolating the target building. We propose an RGB-M masking strategy that appends the building footprint mask as a fourth input channel, providing a soft spatial prior that outperforms hard cropping across both modalities. The Perceiver IO fusion model improves over all other fusion strategies and yields substantial per-class gains for attributes visible from street level (e.g., +11.3 AP for slate, +1.3 AP for dormers), though the satellite-only baseline retains a slight advantage in macro-averaged mAP for classes that are predominantly visible from above. These results establish a scalable, flexible architecture for multi-modal building inspection that can accommodate heterogeneous inputs and multiple output tasks.

Related papers

• Joint Instance Segmentation and Geometric Attribute Regression for Roof Structures in Aerial Imagery [18.60580673640467]
  We present a method for jointly predicting instance-level roof segment masks from a single aerial orthophoto.<n>We introduce two key innovations: a conditional azimuth loss that suppresses supervision for flat roof segments, and a log-normalized height representation.<n>Our method achieves a mean absolute error of approximately 4 degrees for roof slope, 7 degrees for azimuth, and 1 meter for building height, with an instance segmentation AP$_50$ of 0.566.
  arXiv  Detail & Related papers  (2026-05-25T22:37:11Z)
• CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval [54.15776146365823]
  Composed Image Retrieval (CIR) enables users to search for target images using both a reference image and manipulation text.<n>We propose CSMCIR, a unified representation framework that achieves efficient query-target alignment through three synergistic components.
  arXiv  Detail & Related papers  (2026-01-07T09:21:38Z)
• Rethinking Efficient Hierarchical Mixing Architecture for Low-light RAW Image Enhancement [70.94252289772685]
  We introduce a Hierarchical Mixing Architecture (HiMA) for efficient low-light image signal processing (ISP)<n>HiMA leverages the complementary strengths of Transformer and Mamba modules to handle features at large and small scales.<n>To address uneven illumination with strong local variations, we propose Local Distribution Adjustment (LoDA)<n>In addition, to fully exploit the denoised outputs from the first stage, we design a Multi-prior Fusion (MPF) module.
  arXiv  Detail & Related papers  (2025-10-17T10:09:38Z)
• Hierarchical LoRA MoE for Efficient CTR Model Scaling [56.608809143548946]
  HiLoMoE is a hierarchical LoRA MoE framework that enables holistic scaling in a parameter-efficient manner.<n>Unlike conventional stacking, HiLoMoE routes based on prior layer scores rather than outputs, allowing all layers to execute in parallel.
  arXiv  Detail & Related papers  (2025-10-12T03:54:11Z)
• CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbones [0.7448254811651417]
  ViT backbones adopt spatial architectural designs, such as window attention, relative positional embeddings in SAM, and RoPE in DINOv3.<n>We introduce a simple yet effective token merging method that maintains spatial integrity, enabling seamless compatibility with spatial architectures.<n>Our approach employs (i)a 2D reduction strategy to enforce structured token layouts, (ii)a spatial-aware merging algorithm that maintains relative token positions, and (iii)a novel max-magnitude-per-dimension token representation.
  arXiv  Detail & Related papers  (2025-09-26T01:59:29Z)
• OpenFACADES: An Open Framework for Architectural Caption and Attribute Data Enrichment via Street View Imagery [9.111872424676617]
  This study bridges the gaps by introducing OpenFACADES, an open framework that leverages crowdsourced data to enrich building profiles.<n>We automate the detection of building facades in panoramic imagery and tailor a reprojection approach to convert objects into holistic perspective views.<n>Third, we introduce an innovative approach that harnesses and investigates the capabilities of open-source large vision-language models (VLMs) for multi-attribute prediction and open-vocabulary captioning.
  arXiv  Detail & Related papers  (2025-04-01T08:20:13Z)
• Enhancing DeepLabV3+ to Fuse Aerial and Satellite Images for Semantic Segmentation [3.508894670581109]
  We introduce a new transposed conventional layers block for upsampling a second entry to fuse it with high level features.<n>This block is designed to amplify and integrate information from satellite images, thereby enriching the segmentation process.<n>For experiments, we used the LandCover.ai dataset for aerial images, alongside the corresponding dataset sourced from Sentinel 2 data.
  arXiv  Detail & Related papers  (2025-03-28T23:07:39Z)
• SOOD++: Leveraging Unlabeled Data to Boost Oriented Object Detection [68.18620488664187]
  We propose a simple yet effective Semi-supervised Oriented Object Detection method termed SOOD++.<n> Specifically, we observe that objects from aerial images usually have arbitrary orientations, small scales, and dense distribution.<n>Extensive experiments conducted on various oriented object under various labeled settings demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-07-01T07:03:51Z)
• ClusterFormer: Clustering As A Universal Visual Learner [80.79669078819562]
  CLUSTERFORMER is a universal vision model based on the CLUSTERing paradigm with TransFORMER. It is capable of tackling heterogeneous vision tasks with varying levels of clustering granularity. For its efficacy, we hope our work can catalyze a paradigm shift in universal models in computer vision.
  arXiv  Detail & Related papers  (2023-09-22T22:12:30Z)
• Fine-grained building roof instance segmentation based on domain adapted pretraining and composite dual-backbone [13.09940764764909]
  We propose a framework to fulfill semantic interpretation of individual buildings with high-resolution optical satellite imagery. Specifically, the leveraged domain adapted pretraining strategy and composite dual-backbone greatly facilitates the discnative feature learning. Experiment results show that our approach ranks in the first place of the 2023 IEEE GRSS Data Fusion Contest.
  arXiv  Detail & Related papers  (2023-08-10T05:54:57Z)
• A Unified Transformer Framework for Group-based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection [59.21990697929617]
  Humans tend to mine objects by learning from a group of images or several frames of video since we live in a dynamic world. Previous approaches design different networks on similar tasks separately, and they are difficult to apply to each other. We introduce a unified framework to tackle these issues, term as UFO (UnifiedObject Framework for Co-Object Framework)
  arXiv  Detail & Related papers  (2022-03-09T13:35:19Z)
• FloorLevel-Net: Recognizing Floor-Level Lines with Height-Attention-Guided Multi-task Learning [49.30194762653723]
  This work tackles the problem of locating floor-level lines in street-view images, using a supervised deep learning approach. We first compile a new dataset and develop a new data augmentation scheme to synthesize training samples. Next, we design FloorLevel-Net, a multi-task learning network that associates explicit features of building facades and implicit floor-level lines.
  arXiv  Detail & Related papers  (2021-07-06T08:17:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.