A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models

• URL: http://arxiv.org/abs/2411.16407v1
• Date: Mon, 25 Nov 2024 14:12:24 GMT
• Title: A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models
• Authors: Manuel Schwonberg, Claus Werner, Hanno Gottschalk, Carsten Meyer,
• Abstract summary: Domain shifts are one of the major challenges in deep learning based computer vision. UDA methods have emerged which adapt a model to a new target domain by only using unlabeled data of that domain. Recent vision-language models have demonstrated strong generalization capabilities which may facilitate domain adaptation. We show that replacing the encoder of existing UDA methods by a vision-language pre-trained encoder can result in significant performance improvements.
• Score: 1.2499537119440245
• License:
• Abstract: Despite the recent progress in deep learning based computer vision, domain shifts are still one of the major challenges. Semantic segmentation for autonomous driving faces a wide range of domain shifts, e.g. caused by changing weather conditions, new geolocations and the frequent use of synthetic data in model training. Unsupervised domain adaptation (UDA) methods have emerged which adapt a model to a new target domain by only using unlabeled data of that domain. The variety of UDA methods is large but all of them use ImageNet pre-trained models. Recently, vision-language models have demonstrated strong generalization capabilities which may facilitate domain adaptation. We show that simply replacing the encoder of existing UDA methods like DACS by a vision-language pre-trained encoder can result in significant performance improvements of up to 10.0% mIoU on the GTA5-to-Cityscapes domain shift. For the generalization performance to unseen domains, the newly employed vision-language pre-trained encoder provides a gain of up to 13.7% mIoU across three unseen datasets. However, we find that not all UDA methods can be easily paired with the new encoder and that the UDA performance does not always likewise transfer into generalization performance. Finally, we perform our experiments on an adverse weather condition domain shift to further verify our findings on a pure real-to-real domain shift.

Related papers

• Disentangling Masked Autoencoders for Unsupervised Domain Generalization [57.56744870106124]
  Unsupervised domain generalization is fast gaining attention but is still far from well-studied. Disentangled Masked Auto (DisMAE) aims to discover the disentangled representations that faithfully reveal intrinsic features. DisMAE co-trains the asymmetric dual-branch architecture with semantic and lightweight variation encoders.
  arXiv  Detail & Related papers  (2024-07-10T11:11:36Z)
• Grounding Stylistic Domain Generalization with Quantitative Domain Shift Measures and Synthetic Scene Images [63.58800688320182]
  Domain Generalization is a challenging task in machine learning. Current methodology lacks quantitative understanding about shifts in stylistic domain. We introduce a new DG paradigm to address these risks.
  arXiv  Detail & Related papers  (2024-05-24T22:13:31Z)
• ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation [48.039156140237615]
  A Continual Test-Time Adaptation task is proposed to adapt the pre-trained model to continually changing target domains. We design a Visual Domain Adapter (ViDA) for CTTA, explicitly handling both domain-specific and domain-shared knowledge. Our proposed method achieves state-of-the-art performance in both classification and segmentation CTTA tasks.
  arXiv  Detail & Related papers  (2023-06-07T11:18:53Z)
• One-Shot Domain Adaptive and Generalizable Semantic Segmentation with Class-Aware Cross-Domain Transformers [96.51828911883456]
  Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data. Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation. We explore the one-shot unsupervised sim-to-real domain adaptation (OSUDA) and generalization problem, where only one real-world data sample is available.
  arXiv  Detail & Related papers  (2022-12-14T15:54:15Z)
• The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization [10.274423413222763]
  Domain adaptation is crucial to adapt a learned model to new scenarios, such as domain shifts or changing data distributions. Current approaches usually require a large amount of labeled or unlabeled data from the shifted domain. We propose Dynamic Unsupervised Adaptation (DUA) to overcome this problem.
  arXiv  Detail & Related papers  (2021-12-01T12:43:41Z)
• VisDA-2021 Competition Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data [64.91713686654805]
  The Visual Domain Adaptation (VisDA) 2021 competition tests models' ability to adapt to novel test distributions. We will evaluate adaptation to novel viewpoints, backgrounds, modalities and degradation in quality. Performance will be measured using a rigorous protocol, comparing to state-of-the-art domain adaptation methods.
  arXiv  Detail & Related papers  (2021-07-23T03:21:51Z)
• Semi-supervised Meta-learning with Disentanglement for Domain-generalised Medical Image Segmentation [15.351113774542839]
  Generalising models to new data from new centres (termed here domains) remains a challenge. We propose a novel semi-supervised meta-learning framework with disentanglement. We show that the proposed method is robust on different segmentation tasks and achieves state-of-the-art generalisation performance on two public benchmarks.
  arXiv  Detail & Related papers  (2021-06-24T19:50:07Z)
• Knowledge Distillation for BERT Unsupervised Domain Adaptation [2.969705152497174]
  A pre-trained language model, BERT, has brought significant performance improvements across a range of natural language processing tasks. We propose a simple but effective unsupervised domain adaptation method, adversarial adaptation with distillation (AAD) We evaluate our approach in the task of cross-domain sentiment classification on 30 domain pairs.
  arXiv  Detail & Related papers  (2020-10-22T06:51:24Z)
• A Review of Single-Source Deep Unsupervised Visual Domain Adaptation [81.07994783143533]
  Large-scale labeled training datasets have enabled deep neural networks to excel across a wide range of benchmark vision tasks. In many applications, it is prohibitively expensive and time-consuming to obtain large quantities of labeled data. To cope with limited labeled training data, many have attempted to directly apply models trained on a large-scale labeled source domain to another sparsely labeled or unlabeled target domain.
  arXiv  Detail & Related papers  (2020-09-01T00:06:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.