A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods

• URL: http://arxiv.org/abs/2210.01210v1
• Date: Mon, 3 Oct 2022 20:01:29 GMT
• Title: A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
• Abstract summary: Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones. In this work, we consider the Partial Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain. We evaluate 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies.
• Score: 11.63848069914184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones. In this work, we consider the Partial Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain. Most successful algorithms use model selection strategies that rely on target labels to find the best hyper-parameters and/or models along training. However, these strategies violate the main assumption in PDA: only unlabeled target domain samples are available. Moreover, there are also inconsistencies in the experimental settings - architecture, hyper-parameter tuning, number of runs - yielding unfair comparisons. The main goal of this work is to provide a realistic evaluation of PDA methods with the different model selection strategies under a consistent evaluation protocol. We evaluate 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. Our two main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points; (ii) only one method and model selection pair performs well on both datasets. Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.

Related papers

• Domain-Division based Progressive Learning for Source-Free Domain Adaptation [73.44692421206774]
  Source-free domain adaptation requires only a source pre-trained model and an unlabeled target domain.<n>We propose a domain-division based progressive learning method named DPL.<n>Our approach consists of two alternating stages, each beginning with the division of the target domain into easy-to-adapt and hard-to-adapt.
  arXiv  Detail & Related papers  (2026-07-31T09:21:56Z)
• PiPa++: Towards Unification of Domain Adaptive Semantic Segmentation via Self-supervised Learning [34.786268652516355]
  Unsupervised domain adaptive segmentation aims to improve the segmentation accuracy of models on target domains without relying on labeled data from those domains. It seeks to align the feature representations of the source domain (where labeled data is available) and the target domain (where only unlabeled data is present)
  arXiv  Detail & Related papers  (2024-07-24T08:53:29Z)
• Self-Paced Learning for Open-Set Domain Adaptation [50.620824701934]
  Traditional domain adaptation methods presume that the classes in the source and target domains are identical. Open-set domain adaptation (OSDA) addresses this limitation by allowing previously unseen classes in the target domain. We propose a novel framework based on self-paced learning to distinguish common and unknown class samples.
  arXiv  Detail & Related papers  (2023-03-10T14:11:09Z)
• Distributional Shift Adaptation using Domain-Specific Features [41.91388601229745]
  In open-world scenarios, streaming big data can be Out-Of-Distribution (OOD) We propose a simple yet effective approach that relies on correlations in general regardless of whether the features are invariant or not. Our approach uses the most confidently predicted samples identified by an OOD base model to train a new model that effectively adapts to the target domain.
  arXiv  Detail & Related papers  (2022-11-09T04:16:21Z)
• PointInst3D: Segmenting 3D Instances by Points [136.7261709896713]
  We propose a fully-convolutional 3D point cloud instance segmentation method that works in a per-point prediction fashion. We find the key to its success is assigning a suitable target to each sampled point. Our approach achieves promising results on both ScanNet and S3DIS benchmarks.
  arXiv  Detail & Related papers  (2022-04-25T02:41:46Z)
• BMD: A General Class-balanced Multicentric Dynamic Prototype Strategy for Source-free Domain Adaptation [74.93176783541332]
  Source-free Domain Adaptation (SFDA) aims to adapt a pre-trained source model to the unlabeled target domain without accessing the well-labeled source data. To make up for the absence of source data, most existing methods introduced feature prototype based pseudo-labeling strategies. We propose a general class-Balanced Multicentric Dynamic prototype strategy for the SFDA task.
  arXiv  Detail & Related papers  (2022-04-06T13:23:02Z)
• Seeking Similarities over Differences: Similarity-based Domain Alignment for Adaptive Object Detection [86.98573522894961]
  We propose a framework that generalizes the components commonly used by Unsupervised Domain Adaptation (UDA) algorithms for detection. Specifically, we propose a novel UDA algorithm, ViSGA, that leverages the best design choices and introduces a simple but effective method to aggregate features at instance-level. We show that both similarity-based grouping and adversarial training allows our model to focus on coarsely aligning feature groups, without being forced to match all instances across loosely aligned domains.
  arXiv  Detail & Related papers  (2021-10-04T13:09:56Z)
• On Universal Black-Box Domain Adaptation [53.7611757926922]
  We study an arguably least restrictive setting of domain adaptation in a sense of practical deployment. Only the interface of source model is available to the target domain, and where the label-space relations between the two domains are allowed to be different and unknown. We propose to unify them into a self-training framework, regularized by consistency of predictions in local neighborhoods of target samples.
  arXiv  Detail & Related papers  (2021-04-10T02:21:09Z)
• OVANet: One-vs-All Network for Universal Domain Adaptation [78.86047802107025]
  Existing methods manually set a threshold to reject unknown samples based on validation or a pre-defined ratio of unknown samples. We propose a method to learn the threshold using source samples and to adapt it to the target domain. Our idea is that a minimum inter-class distance in the source domain should be a good threshold to decide between known or unknown in the target.
  arXiv  Detail & Related papers  (2021-04-07T18:36:31Z)
• Domain Generalization via Inference-time Label-Preserving Target Projections [6.97174182373899]
  Generalization of machine learning models trained on a set of source domains on unseen target domains with different statistics is a challenging problem. We propose a method that effectively uses the target sample during inference beyond mere classification. We demonstrate that our method outperforms the state-of-the-art Domain Generalization methods on multiple datasets and tasks.
  arXiv  Detail & Related papers  (2021-03-01T17:10:38Z)
• A simple baseline for domain adaptation using rotation prediction [17.539027866457673]
  The goal is to adapt a model trained in one domain to another domain with scarce annotated data. We propose a simple yet effective method based on self-supervised learning. Our simple method achieves state-of-the-art results on semi-supervised domain adaptation on DomainNet dataset.
  arXiv  Detail & Related papers  (2019-12-26T17:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.