ControlUDA: Controllable Diffusion-assisted Unsupervised Domain Adaptation for Cross-Weather Semantic Segmentation

• URL: http://arxiv.org/abs/2402.06446v1
• Date: Fri, 9 Feb 2024 14:48:20 GMT
• Title: ControlUDA: Controllable Diffusion-assisted Unsupervised Domain Adaptation for Cross-Weather Semantic Segmentation
• Authors: Fengyi Shen, Li Zhou, Kagan Kucukaytekin, Ziyuan Liu, He Wang, Alois Knoll
• Abstract summary: ControlUDA is a diffusion-assisted framework tailored for UDA segmentation under adverse weather conditions. It first leverages target prior from a pre-trained segmentor for tuning the DM, compensating the missing target domain labels. It also contains UDAControlNet, a condition-fused multi-scale and prompt-enhanced network targeted at high-fidelity data generation in adverse weathers.
• Score: 14.407346832155042
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Data generation is recognized as a potent strategy for unsupervised domain adaptation (UDA) pertaining semantic segmentation in adverse weathers. Nevertheless, these adverse weather scenarios encompass multiple possibilities, and high-fidelity data synthesis with controllable weather is under-researched in previous UDA works. The recent strides in large-scale text-to-image diffusion models (DM) have ushered in a novel avenue for research, enabling the generation of realistic images conditioned on semantic labels. This capability proves instrumental for cross-domain data synthesis from source to target domain owing to their shared label space. Thus, source domain labels can be paired with those generated pseudo target data for training UDA. However, from the UDA perspective, there exists several challenges for DM training: (i) ground-truth labels from target domain are missing; (ii) the prompt generator may produce vague or noisy descriptions of images from adverse weathers; (iii) existing arts often struggle to well handle the complex scene structure and geometry of urban scenes when conditioned only on semantic labels. To tackle the above issues, we propose ControlUDA, a diffusion-assisted framework tailored for UDA segmentation under adverse weather conditions. It first leverages target prior from a pre-trained segmentor for tuning the DM, compensating the missing target domain labels; It also contains UDAControlNet, a condition-fused multi-scale and prompt-enhanced network targeted at high-fidelity data generation in adverse weathers. Training UDA with our generated data brings the model performances to a new milestone (72.0 mIoU) on the popular Cityscapes-to-ACDC benchmark for adverse weathers. Furthermore, ControlUDA helps to achieve good model generalizability on unseen data.

Related papers

• SiamSeg: Self-Training with Contrastive Learning for Unsupervised Domain Adaptation Semantic Segmentation in Remote Sensing [13.549403813487022]
  Unsupervised domain adaptation (UDA) enables models to learn from unlabeled target domain data while leveraging labeled source domain data. We propose integrating contrastive learning into UDA, enhancing the model's ability to capture semantic information in the target domain. Our method, SimSeg, outperforms existing approaches, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2024-10-17T11:59:39Z)
• Diffuse-UDA: Addressing Unsupervised Domain Adaptation in Medical Image Segmentation with Appearance and Structure Aligned Diffusion Models [31.006056670998852]
  The scarcity and complexity of voxel-level annotations in 3D medical imaging present significant challenges. This disparity affects the fairness of artificial intelligence algorithms in healthcare. We introduce Diffuse-UDA, a novel method leveraging diffusion models to tackle Unsupervised Domain Adaptation (UDA) in medical image segmentation.
  arXiv  Detail & Related papers  (2024-08-12T08:21:04Z)
• Disentangling Masked Autoencoders for Unsupervised Domain Generalization [57.56744870106124]
  Unsupervised domain generalization is fast gaining attention but is still far from well-studied. Disentangled Masked Auto (DisMAE) aims to discover the disentangled representations that faithfully reveal intrinsic features. DisMAE co-trains the asymmetric dual-branch architecture with semantic and lightweight variation encoders.
  arXiv  Detail & Related papers  (2024-07-10T11:11:36Z)
• DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series [25.434379659643707]
  In time series anomaly detection, the scarcity of labeled data poses a challenge to the development of accurate models. We propose a novel Domain Contrastive learning model for Anomaly Detection in time series (DACAD) Our model employs supervised contrastive loss for the source domain and self-supervised contrastive triplet loss for the target domain.
  arXiv  Detail & Related papers  (2024-04-17T11:20:14Z)
• Unified Domain Adaptive Semantic Segmentation [96.74199626935294]
  Unsupervised Adaptive Domain Semantic (UDA-SS) aims to transfer the supervision from a labeled source domain to an unlabeled target domain. We propose a Quad-directional Mixup (QuadMix) method, characterized by tackling distinct point attributes and feature inconsistencies. Our method outperforms the state-of-the-art works by large margins on four challenging UDA-SS benchmarks.
  arXiv  Detail & Related papers  (2023-11-22T09:18:49Z)
• Source-Free Domain Adaptation for Medical Image Segmentation via Prototype-Anchored Feature Alignment and Contrastive Learning [57.43322536718131]
  We present a two-stage source-free domain adaptation (SFDA) framework for medical image segmentation. In the prototype-anchored feature alignment stage, we first utilize the weights of the pre-trained pixel-wise classifier as source prototypes. Then, we introduce the bi-directional transport to align the target features with class prototypes by minimizing its expected cost.
  arXiv  Detail & Related papers  (2023-07-19T06:07:12Z)
• Enhancing Visual Domain Adaptation with Source Preparation [5.287588907230967]
  Domain Adaptation techniques fail to consider the characteristics of the source domain itself. We propose Source Preparation (SP), a method to mitigate source domain biases. We show that SP enhances UDA across a range of visual domains, with improvements up to 40.64% in mIoU over baseline.
  arXiv  Detail & Related papers  (2023-06-16T18:56:44Z)
• One-Shot Domain Adaptive and Generalizable Semantic Segmentation with Class-Aware Cross-Domain Transformers [96.51828911883456]
  Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data. Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation. We explore the one-shot unsupervised sim-to-real domain adaptation (OSUDA) and generalization problem, where only one real-world data sample is available.
  arXiv  Detail & Related papers  (2022-12-14T15:54:15Z)
• An Unsupervised Domain Adaptive Approach for Multimodal 2D Object Detection in Adverse Weather Conditions [5.217255784808035]
  We propose an unsupervised domain adaptation framework to bridge the domain gap between source and target domains. We use a data augmentation scheme that simulates weather distortions to add domain confusion and prevent overfitting on the source data. Experiments performed on the DENSE dataset show that our method can substantially alleviate the domain gap.
  arXiv  Detail & Related papers  (2022-03-07T18:10:40Z)
• Stagewise Unsupervised Domain Adaptation with Adversarial Self-Training for Road Segmentation of Remote Sensing Images [93.50240389540252]
  Road segmentation from remote sensing images is a challenging task with wide ranges of application potentials. We propose a novel stagewise domain adaptation model called RoadDA to address the domain shift (DS) issue in this field. Experiment results on two benchmarks demonstrate that RoadDA can efficiently reduce the domain gap and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-28T09:29:14Z)
• Semi-Supervised Domain Adaptation with Prototypical Alignment and Consistency Learning [86.6929930921905]
  This paper studies how much it can help address domain shifts if we further have a few target samples labeled. To explore the full potential of landmarks, we incorporate a prototypical alignment (PA) module which calculates a target prototype for each class from the landmarks. Specifically, we severely perturb the labeled images, making PA non-trivial to achieve and thus promoting model generalizability.
  arXiv  Detail & Related papers  (2021-04-19T08:46:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.