DUDA: Distilled Unsupervised Domain Adaptation for Lightweight Semantic Segmentation

• URL: http://arxiv.org/abs/2504.09814v1
• Date: Mon, 14 Apr 2025 02:30:18 GMT
• Title: DUDA: Distilled Unsupervised Domain Adaptation for Lightweight Semantic Segmentation
• Authors: Beomseok Kang, Niluthpol Chowdhury Mithun, Abhinav Rajvanshi, Han-Pang Chiu, Supun Samarasekera,
• Abstract summary: Unsupervised Domain Adaptation (UDA) is essential for enabling semantic segmentation in new domains without requiring costly pixel-wise annotations.<n>We propose Distilled Unsupervised Domain Adaptation (DUDA), a novel framework that combines EMA-based self-training with knowledge distillation.<n>Our method employs an auxiliary student network to bridge the architectural gap between heavyweight and lightweight models for EMA-based updates.
• Score: 9.568820012635355
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unsupervised Domain Adaptation (UDA) is essential for enabling semantic segmentation in new domains without requiring costly pixel-wise annotations. State-of-the-art (SOTA) UDA methods primarily use self-training with architecturally identical teacher and student networks, relying on Exponential Moving Average (EMA) updates. However, these approaches face substantial performance degradation with lightweight models due to inherent architectural inflexibility leading to low-quality pseudo-labels. To address this, we propose Distilled Unsupervised Domain Adaptation (DUDA), a novel framework that combines EMA-based self-training with knowledge distillation (KD). Our method employs an auxiliary student network to bridge the architectural gap between heavyweight and lightweight models for EMA-based updates, resulting in improved pseudo-label quality. DUDA employs a strategic fusion of UDA and KD, incorporating innovative elements such as gradual distillation from large to small networks, inconsistency loss prioritizing poorly adapted classes, and learning with multiple teachers. Extensive experiments across four UDA benchmarks demonstrate DUDA's superiority in achieving SOTA performance with lightweight models, often surpassing the performance of heavyweight models from other approaches.

Related papers

• One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation [69.65734716679925]
  Knowledge distillation has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. Most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family. We propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures.
  arXiv  Detail & Related papers  (2023-10-30T11:13:02Z)
• Instance-aware Model Ensemble With Distillation For Unsupervised Domain Adaptation [28.79286984013436]
  We propose a novel framework, namely Instance aware Model Ensemble With Distillation, IMED. IMED fuses multiple UDA component models adaptively according to different instances and distills these components into a small model. We show the superiority of the model based on IMED to the state of the art methods under the comparable computation cost.
  arXiv  Detail & Related papers  (2022-11-15T12:53:23Z)
• Exploring Adversarially Robust Training for Unsupervised Domain Adaptation [71.94264837503135]
  Unsupervised Domain Adaptation (UDA) methods aim to transfer knowledge from a labeled source domain to an unlabeled target domain. This paper explores how to enhance the unlabeled data robustness via AT while learning domain-invariant features for UDA. We propose a novel Adversarially Robust Training method for UDA accordingly, referred to as ARTUDA.
  arXiv  Detail & Related papers  (2022-02-18T17:05:19Z)
• Revisiting Deep Subspace Alignment for Unsupervised Domain Adaptation [42.16718847243166]
  Unsupervised domain adaptation (UDA) aims to transfer and adapt knowledge from a labeled source domain to an unlabeled target domain. Traditionally, subspace-based methods form an important class of solutions to this problem. This paper revisits the use of subspace alignment for UDA and proposes a novel adaptation algorithm that consistently leads to improved generalization.
  arXiv  Detail & Related papers  (2022-01-05T20:16:38Z)
• Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching [82.71578668091914]
  This paper establishes a compact learning framework that embeds the classification and mask-refinement components into a unified deep model. We propose a novel alternative self-dual teaching (ASDT) mechanism to encourage high-quality knowledge interaction.
  arXiv  Detail & Related papers  (2021-12-17T11:56:56Z)
• DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation [99.88539409432916]
  We study the unsupervised domain adaptation (UDA) process. We propose a novel UDA method, DAFormer, based on the benchmark results. DAFormer significantly improves the state-of-the-art performance by 10.8 mIoU for GTA->Cityscapes and 5.4 mIoU for Synthia->Cityscapes.
  arXiv  Detail & Related papers  (2021-11-29T19:00:46Z)
• Adversarial Robustness for Unsupervised Domain Adaptation [48.51898925429575]
  In this work, we leverage intermediate representations learned by multiple robust ImageNet models to improve the robustness of UDA models. Our method works by aligning the features of the UDA model with the robust features learned by ImageNet pre-trained models along with domain adaptation training.
  arXiv  Detail & Related papers  (2021-09-02T13:45:01Z)
• A New Bidirectional Unsupervised Domain Adaptation Segmentation Framework [27.13101555533594]
  unsupervised domain adaptation (UDA) techniques are proposed to bridge the gap between different domains. In this paper, we propose a bidirectional UDA framework based on disentangled representation learning for equally competent two-way UDA performances.
  arXiv  Detail & Related papers  (2021-08-18T05:25:11Z)
• Class-Incremental Domain Adaptation [56.72064953133832]
  We introduce a practical Domain Adaptation (DA) paradigm called Class-Incremental Domain Adaptation (CIDA) Existing DA methods tackle domain-shift but are unsuitable for learning novel target-domain classes. Our approach yields superior performance as compared to both DA and CI methods in the CIDA paradigm.
  arXiv  Detail & Related papers  (2020-08-04T07:55:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.