Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels

• URL: http://arxiv.org/abs/2607.10841v1
• Date: Sun, 12 Jul 2026 17:07:55 GMT
• Title: Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels
• Abstract summary: Supervised learning for image segmentation typically requires spatially aligned image and label sets.<n> misaligned labels can significantly deteriorate the performance of the learned models.<n>We propose a novel approach for training on misaligned labels, where we simultaneously learn the label alignment.
• Score: 15.14055526507309
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Supervised learning for image segmentation typically requires spatially aligned image and label sets. When images and labels originate from different sources, the pairing may be misaligned, which can significantly deteriorate the performance of the learned models. This is especially common in remote sensing, when aerial or satellite images are co-registered with labels from another source (e.g., OpenStreetMap). In this work, we propose a novel approach for training on misaligned labels, where we simultaneously learn the label alignment. Our align and segment (AnS) approach builds on the spatial transformer module to transform the misaligned labels using an affine transformation to provide a better learning target for a canonical semantic segmentation network. We prevent shortcut learning of misaligned labels in these semantic segmentation networks through a self-supervised regularization loss and show that it is complementary to data augmentation, especially for systematically misaligned training data. A decisive characteristic of our AnS approach is that it learns without requiring any golden labels. We experimentally show on both synthetic and real-world data from different cities that our approach enables high-quality building segmentation and precise label-image alignment at the same time. Code and derived datasets are available at https://github.com/venkanna37/align-and-segment

Related papers

• Zero-Shot Pseudo Labels Generation Using SAM and CLIP for Semi-Supervised Semantic Segmentation [0.0]
  We propose a method to train a semantic segmentation model using images with annotated labels and pseudo labels.<n>The accuracy of the model depends on the quality of the pseudo labels and the amount of data with annotated labels.<n>The effectiveness of the proposed method is demonstrated through the experiments using the public datasets: PASCAL and MS COCO.
  arXiv  Detail & Related papers  (2025-05-26T11:31:13Z)
• Dual-level Interaction for Domain Adaptive Semantic Segmentation [0.0]
  We propose a dual-level interaction for domain adaptation (DIDA) in semantic segmentation. Explicitly, we encourage the different augmented views of the same pixel to have similar class prediction. Our method outperforms the state-of-the-art by a notable margin, especially on confusing and long-tailed classes.
  arXiv  Detail & Related papers  (2023-07-16T07:51:18Z)
• Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation [58.03255076119459]
  We address the task of weakly-supervised few-shot image classification and segmentation, by leveraging a Vision Transformer (ViT) Our proposed method takes token representations from the self-supervised ViT and leverages their correlations, via self-attention, to produce classification and segmentation predictions. Experiments on Pascal-5i and COCO-20i demonstrate significant performance gains in a variety of supervision settings.
  arXiv  Detail & Related papers  (2023-07-07T06:16:43Z)
• G2NetPL: Generic Game-Theoretic Network for Partial-Label Image Classification [14.82038002764209]
  Multi-label image classification aims to predict all possible labels in an image. Existing works on partial-label learning focus on the case where each training image is labeled with only a subset of its positive/negative labels. This paper proposes an end-to-end Generic Game-theoretic Network (G2NetPL) for partial-label learning.
  arXiv  Detail & Related papers  (2022-10-20T17:59:21Z)
• Language-driven Semantic Segmentation [88.21498323896475]
  We present LSeg, a novel model for language-driven semantic image segmentation. We use a text encoder to compute embeddings of descriptive input labels. The encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class.
  arXiv  Detail & Related papers  (2022-01-10T18:59:10Z)
• Structured Semantic Transfer for Multi-Label Recognition with Partial Labels [85.6967666661044]
  We propose a structured semantic transfer (SST) framework that enables training multi-label recognition models with partial labels. The framework consists of two complementary transfer modules that explore within-image and cross-image semantic correlations. Experiments on the Microsoft COCO, Visual Genome and Pascal VOC datasets show that the proposed SST framework obtains superior performance over current state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2021-12-21T02:15:01Z)
• Joining datasets via data augmentation in the label space for neural networks [6.036150783745836]
  We propose a new technique leveraging artificially created knowledge graph, recurrent neural networks and policy gradient that successfully achieve the dataset joining in the label space. Empirical results on both image and text classification justify the validity of our approach.
  arXiv  Detail & Related papers  (2021-06-17T06:08:11Z)
• A Closer Look at Self-training for Zero-Label Semantic Segmentation [53.4488444382874]
  Being able to segment unseen classes not observed during training is an important technical challenge in deep learning. Prior zero-label semantic segmentation works approach this task by learning visual-semantic embeddings or generative models. We propose a consistency regularizer to filter out noisy pseudo-labels by taking the intersections of the pseudo-labels generated from different augmentations of the same image.
  arXiv  Detail & Related papers  (2021-04-21T14:34:33Z)
• PseudoSeg: Designing Pseudo Labels for Semantic Segmentation [78.35515004654553]
  We present a re-design of pseudo-labeling to generate structured pseudo labels for training with unlabeled or weakly-labeled data. We demonstrate the effectiveness of the proposed pseudo-labeling strategy in both low-data and high-data regimes.
  arXiv  Detail & Related papers  (2020-10-19T17:59:30Z)
• Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition [75.44233392355711]
  KGGR framework exploits prior knowledge of statistical label correlations with deep neural networks. It first builds a structured knowledge graph to correlate different labels based on statistical label co-occurrence. Then, it introduces the label semantics to guide learning semantic-specific features. It exploits a graph propagation network to explore graph node interactions.
  arXiv  Detail & Related papers  (2020-09-20T15:05:29Z)
• Multi-label Zero-shot Classification by Learning to Transfer from External Knowledge [36.04579549557464]
  Multi-label zero-shot classification aims to predict multiple unseen class labels for an input image. This paper introduces a novel multi-label zero-shot classification framework by learning to transfer from external knowledge.
  arXiv  Detail & Related papers  (2020-07-30T17:26:46Z)
• Reconstruction Regularized Deep Metric Learning for Multi-label Image Classification [39.055689258395624]
  We present a novel deep metric learning method to tackle the multi-label image classification problem. Our model can be trained in an end-to-end manner.
  arXiv  Detail & Related papers  (2020-07-27T13:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.