RLSbench: Domain Adaptation Under Relaxed Label Shift

• URL: http://arxiv.org/abs/2302.03020v2
• Date: Mon, 5 Jun 2023 13:55:19 GMT
• Title: RLSbench: Domain Adaptation Under Relaxed Label Shift
• Authors: Saurabh Garg, Nick Erickson, James Sharpnack, Alex Smola, Sivaraman Balakrishnan, Zachary C. Lipton
• Abstract summary: We introduce RLSbench, a large-scale benchmark for relaxed label shift. We assess 13 popular domain adaptation methods, demonstrating more widespread failures under label proportion shifts than were previously known. We develop an effective two-step meta-algorithm that is compatible with most domain adaptations.
• Score: 39.845383643588356
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the emergence of principled methods for domain adaptation under label shift, their sensitivity to shifts in class conditional distributions is precariously under explored. Meanwhile, popular deep domain adaptation heuristics tend to falter when faced with label proportions shifts. While several papers modify these heuristics in attempts to handle label proportions shifts, inconsistencies in evaluation standards, datasets, and baselines make it difficult to gauge the current best practices. In this paper, we introduce RLSbench, a large-scale benchmark for relaxed label shift, consisting of $>$500 distribution shift pairs spanning vision, tabular, and language modalities, with varying label proportions. Unlike existing benchmarks, which primarily focus on shifts in class-conditional $p(x|y)$, our benchmark also focuses on label marginal shifts. First, we assess 13 popular domain adaptation methods, demonstrating more widespread failures under label proportion shifts than were previously known. Next, we develop an effective two-step meta-algorithm that is compatible with most domain adaptation heuristics: (i) pseudo-balance the data at each epoch; and (ii) adjust the final classifier with target label distribution estimate. The meta-algorithm improves existing domain adaptation heuristics under large label proportion shifts, often by 2--10\% accuracy points, while conferring minimal effect ($<$0.5\%) when label proportions do not shift. We hope that these findings and the availability of RLSbench will encourage researchers to rigorously evaluate proposed methods in relaxed label shift settings. Code is publicly available at https://github.com/acmi-lab/RLSbench.

Related papers

• Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation [16.657929958093824]
  Test-time adaptation is an approach to adjust models to a new data distribution during inference. Test-time batch normalization is a simple and popular method that achieved compelling performance on domain shift benchmarks. We propose to tackle this challenge by only selectively adapting channels in a deep network, minimizing drastic adaptation that is sensitive to label shifts.
  arXiv  Detail & Related papers  (2024-02-07T15:41:01Z)
• Generating Unbiased Pseudo-labels via a Theoretically Guaranteed Chebyshev Constraint to Unify Semi-supervised Classification and Regression [57.17120203327993]
  threshold-to-pseudo label process (T2L) in classification uses confidence to determine the quality of label. In nature, regression also requires unbiased methods to generate high-quality labels. We propose a theoretically guaranteed constraint for generating unbiased labels based on Chebyshev's inequality.
  arXiv  Detail & Related papers  (2023-11-03T08:39:35Z)
• Label Shift Adapter for Test-Time Adaptation under Covariate and Label Shifts [48.83127071288469]
  Test-time adaptation (TTA) aims to adapt a pre-trained model to the target domain in a batch-by-batch manner during inference. Most previous TTA approaches assume that both source and target domain datasets have balanced label distribution. We propose a novel label shift adapter that can be incorporated into existing TTA approaches to deal with label shifts effectively.
  arXiv  Detail & Related papers  (2023-08-17T06:37:37Z)
• Domain Adaptation under Open Set Label Shift [39.424134505152544]
  We introduce the problem of domain adaptation under Open Set Label Shift (OSLS) OSLS subsumes domain adaptation under label shift and Positive-Unlabeled (PU) learning. We propose practical methods for both tasks that leverage black-box predictors.
  arXiv  Detail & Related papers  (2022-07-26T17:09:48Z)
• Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning [80.05441565830726]
  This paper addresses imbalanced semi-supervised learning, where heavily biased pseudo-labels can harm the model performance. We propose a general pseudo-labeling framework to address the bias motivated by this observation. We term the novel pseudo-labeling framework for imbalanced SSL as Distribution-Aware Semantics-Oriented (DASO) Pseudo-label.
  arXiv  Detail & Related papers  (2021-06-10T11:58:25Z)
• Rethinking Pseudo Labels for Semi-Supervised Object Detection [84.697097472401]
  We introduce certainty-aware pseudo labels tailored for object detection. We dynamically adjust the thresholds used to generate pseudo labels and reweight loss functions for each category to alleviate the class imbalance problem. Our approach improves supervised baselines by up to 10% AP using only 1-10% labeled data from COCO.
  arXiv  Detail & Related papers  (2021-06-01T01:32:03Z)
• Coping with Label Shift via Distributionally Robust Optimisation [72.80971421083937]
  We propose a model that minimises an objective based on distributionally robust optimisation (DRO) We then design and analyse a gradient descent-proximal mirror ascent algorithm tailored for large-scale problems to optimise the proposed objective.
  arXiv  Detail & Related papers  (2020-10-23T08:33:04Z)
• Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift [20.533804144992207]
  Adversarial learning has demonstrated good performance in the unsupervised domain adaptation setting. We propose a new assumption, generalized label shift ($GLS$), to improve robustness against mismatched label distributions. Our algorithms outperform the base versions, with vast improvements for large label distribution mismatches.
  arXiv  Detail & Related papers  (2020-03-10T00:35:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.