Transferable Mask Transformer: Cross-domain Semantic Segmentation with Region-adaptive Transferability Estimation

• URL: http://arxiv.org/abs/2504.05774v1
• Date: Tue, 08 Apr 2025 07:53:51 GMT
• Title: Transferable Mask Transformer: Cross-domain Semantic Segmentation with Region-adaptive Transferability Estimation
• Authors: Enming Zhang, Zhengyu Li, Yanru Wu, Jingge Wang, Yang Tan, Ruizhe Zhao, Guan Wang, Yang Li,
• Abstract summary: Transferable Mask Transformer (TMT) is a novel region-level adaptation framework for semantic segmentation.<n>TMT aligns cross-domain representations through spatial transferability analysis.<n>TMT achieves an average 2% MIoU improvement over vanilla fine-tuning and a 1.28% increase compared to state-of-the-art baselines.
• Score: 6.355831653594172
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in Vision Transformers (ViTs) have set new benchmarks in semantic segmentation. However, when adapting pretrained ViTs to new target domains, significant performance degradation often occurs due to distribution shifts, resulting in suboptimal global attention. Since self-attention mechanisms are inherently data-driven, they may fail to effectively attend to key objects when source and target domains exhibit differences in texture, scale, or object co-occurrence patterns. While global and patch-level domain adaptation methods provide partial solutions, region-level adaptation with dynamically shaped regions is crucial due to spatial heterogeneity in transferability across different image areas. We present Transferable Mask Transformer (TMT), a novel region-level adaptation framework for semantic segmentation that aligns cross-domain representations through spatial transferability analysis. TMT consists of two key components: (1) An Adaptive Cluster-based Transferability Estimator (ACTE) that dynamically segments images into structurally and semantically coherent regions for localized transferability assessment, and (2) A Transferable Masked Attention (TMA) module that integrates region-specific transferability maps into ViTs' attention mechanisms, prioritizing adaptation in regions with low transferability and high semantic uncertainty. Comprehensive evaluations across 20 cross-domain pairs demonstrate TMT's superiority, achieving an average 2% MIoU improvement over vanilla fine-tuning and a 1.28% increase compared to state-of-the-art baselines. The source code will be publicly available.

Related papers

• TransAdapter: Vision Transformer for Feature-Centric Unsupervised Domain Adaptation [0.3277163122167433]
  Unsupervised Domain Adaptation (UDA) aims to utilize labeled data from a source domain to solve tasks in an unlabeled target domain.<n>Traditional CNN-based methods struggle to fully capture complex domain relationships.<n>We propose a novel UDA approach leveraging the Swin Transformer with three key modules.
  arXiv  Detail & Related papers  (2024-12-05T11:11:39Z)
• Exploring Consistency in Cross-Domain Transformer for Domain Adaptive Semantic Segmentation [51.10389829070684]
  Domain gap can cause discrepancies in self-attention. Due to this gap, the transformer attends to spurious regions or pixels, which deteriorates accuracy on the target domain. We propose adaptation on attention maps with cross-domain attention layers.
  arXiv  Detail & Related papers  (2022-11-27T02:40:33Z)
• UniDAformer: Unified Domain Adaptive Panoptic Segmentation Transformer via Hierarchical Mask Calibration [49.16591283724376]
  We design UniDAformer, a unified domain adaptive panoptic segmentation transformer that is simple but can achieve domain adaptive instance segmentation and semantic segmentation simultaneously within a single network. UniDAformer introduces Hierarchical Mask (HMC) that rectifies inaccurate predictions at the level of regions, superpixels and annotated pixels via online self-training on the fly. It has three unique features: 1) it enables unified domain adaptive panoptic adaptation; 2) it mitigates false predictions and improves domain adaptive panoptic segmentation effectively; 3) it is end-to-end trainable with a much simpler training and inference pipeline.
  arXiv  Detail & Related papers  (2022-06-30T07:32:23Z)
• Smoothing Matters: Momentum Transformer for Domain Adaptive Semantic Segmentation [48.7190017311309]
  We find that straightforwardly applying local ViTs in domain adaptive semantic segmentation does not bring in expected improvement. These high-frequency components make the training of local ViTs very unsmooth and hurt their transferability. In this paper, we introduce a low-pass filtering mechanism, momentum network, to smooth the learning dynamics of target domain features and pseudo labels.
  arXiv  Detail & Related papers  (2022-03-15T15:20:30Z)
• Amplitude Spectrum Transformation for Open Compound Domain Adaptive Semantic Segmentation [62.68759523116924]
  Open compound domain adaptation (OCDA) has emerged as a practical adaptation setting. We propose a novel feature space Amplitude Spectrum Transformation (AST)
  arXiv  Detail & Related papers  (2022-02-09T05:40:34Z)
• Domain Adaptive Semantic Segmentation with Regional Contrastive Consistency Regularization [19.279884432843822]
  We propose a novel and fully end-to-end trainable approach, called regional contrastive consistency regularization (RCCR) for domain adaptive semantic segmentation. Our core idea is to pull the similar regional features extracted from the same location of different images to be closer, and meanwhile push the features from the different locations of the two images to be separated.
  arXiv  Detail & Related papers  (2021-10-11T11:45:00Z)
• Transformer-Based Source-Free Domain Adaptation [134.67078085569017]
  We study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. We propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA.
  arXiv  Detail & Related papers  (2021-05-28T23:06:26Z)
• Cross-Domain Grouping and Alignment for Domain Adaptive Semantic Segmentation [74.3349233035632]
  Existing techniques to adapt semantic segmentation networks across the source and target domains within deep convolutional neural networks (CNNs) do not consider an inter-class variation within the target domain itself or estimated category. We introduce a learnable clustering module, and a novel domain adaptation framework called cross-domain grouping and alignment. Our method consistently boosts the adaptation performance in semantic segmentation, outperforming the state-of-the-arts on various domain adaptation settings.
  arXiv  Detail & Related papers  (2020-12-15T11:36:21Z)
• Contextual-Relation Consistent Domain Adaptation for Semantic Segmentation [44.19436340246248]
  This paper presents an innovative local contextual-relation consistent domain adaptation technique. It aims to achieve local-level consistencies during the global-level alignment. Experiments demonstrate its superior segmentation performance as compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-05T19:00:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.