Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlations

• URL: http://arxiv.org/abs/2211.15646v4
• Date: Wed, 29 Nov 2023 02:42:18 GMT
• Title: Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlations
• Authors: Qingyao Sun (Cornell University), Kevin Murphy (Google DeepMind), Sayna Ebrahimi (Google Cloud AI Research), Alexander D'Amour (Google DeepMind)
• Abstract summary: Changes in the data distribution at test time can have deleterious effects on the performance of predictive models. We propose a test-time label shift correction that adapts to changes in the joint distribution $p(y, z)$ using EM applied to unlabeled samples.
• Score: 44.99833362998488
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Changes in the data distribution at test time can have deleterious effects on the performance of predictive models $p(y|x)$. We consider situations where there are additional meta-data labels (such as group labels), denoted by $z$, that can account for such changes in the distribution. In particular, we assume that the prior distribution $p(y, z)$, which models the dependence between the class label $y$ and the "nuisance" factors $z$, may change across domains, either due to a change in the correlation between these terms, or a change in one of their marginals. However, we assume that the generative model for features $p(x|y,z)$ is invariant across domains. We note that this corresponds to an expanded version of the widely used "label shift" assumption, where the labels now also include the nuisance factors $z$. Based on this observation, we propose a test-time label shift correction that adapts to changes in the joint distribution $p(y, z)$ using EM applied to unlabeled samples from the target domain distribution, $p_t(x)$. Importantly, we are able to avoid fitting a generative model $p(x|y, z)$, and merely need to reweight the outputs of a discriminative model $p_s(y, z|x)$ trained on the source distribution. We evaluate our method, which we call "Test-Time Label-Shift Adaptation" (TTLSA), on several standard image and text datasets, as well as the CheXpert chest X-ray dataset, and show that it improves performance over methods that target invariance to changes in the distribution, as well as baseline empirical risk minimization methods. Code for reproducing experiments is available at https://github.com/nalzok/test-time-label-shift .

Related papers

• Weighted Risk Invariance: Domain Generalization under Invariant Feature Shift [41.60879054101201]
  Learning models whose predictions are invariant under multiple environments is a promising approach. We show that learning invariant models underperform under certain conditions. We propose a practical to implement WRI by learning the correlations $p(X_textinv) and the model parameters simultaneously.
  arXiv  Detail & Related papers  (2024-07-25T23:27:10Z)
• One-Bit Quantization and Sparsification for Multiclass Linear Classification with Strong Regularization [18.427215139020625]
  We show that the best classification is achieved when $f(cdot) = |cdot|2$ and $lambda to infty$. It is often possible to find sparse and one-bit solutions that perform almost as well as one corresponding to $f(cdot) = |cdot|_infty$ in the large $lambda$ regime.
  arXiv  Detail & Related papers  (2024-02-16T06:39:40Z)
• Testing Dependency of Unlabeled Databases [5.384630221560811]
  Two random databases $mathsfXinmathcalXntimes d$ and $mathsfYinmathcalYntimes d$ are statistically dependent or not. We characterize the thresholds at which optimal testing is information-theoretically impossible and possible.
  arXiv  Detail & Related papers  (2023-11-10T05:17:03Z)
• Statistical Learning under Heterogeneous Distribution Shift [71.8393170225794]
  Ground-truth predictor is additive $mathbbE[mathbfz mid mathbfx,mathbfy] = f_star(mathbfx) +g_star(mathbfy)$.
  arXiv  Detail & Related papers  (2023-02-27T16:34:21Z)
• How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm? [73.80001705134147]
  We provide an exact characterization of the expected generalization error (gen-error) for semi-supervised learning (SSL) with pseudo-labeling via the Gibbs algorithm. The gen-error is expressed in terms of the symmetrized KL information between the output hypothesis, the pseudo-labeled dataset, and the labeled dataset.
  arXiv  Detail & Related papers  (2022-10-15T04:11:56Z)
• Bias Mimicking: A Simple Sampling Approach for Bias Mitigation [57.17709477668213]
  We introduce a new class-conditioned sampling method: Bias Mimicking. Bias Mimicking improves underrepresented groups' accuracy of sampling methods by 3% over four benchmarks.
  arXiv  Detail & Related papers  (2022-09-30T17:33:00Z)
• Unsupervised Learning under Latent Label Shift [21.508249151557244]
  We introduce unsupervised learning under Latent Label Shift (LLS) We show that our algorithm can leverage domain information to improve state of the art unsupervised classification methods.
  arXiv  Detail & Related papers  (2022-07-26T20:52:53Z)
• Data fission: splitting a single data point [27.500860533521713]
  In this paper, we offer a more general methodology for achieving such a split in finite samples. We call our method data fission, as an alternative to data splitting, data carving and p-value masking. We exemplify the method on a few applications, such as post-selection inference for trend filtering and other regression problems.
  arXiv  Detail & Related papers  (2021-12-21T10:27:04Z)
• Instance-Dependent Partial Label Learning [69.49681837908511]
  Partial label learning is a typical weakly supervised learning problem. Most existing approaches assume that the incorrect labels in each training example are randomly picked as the candidate labels. In this paper, we consider instance-dependent and assume that each example is associated with a latent label distribution constituted by the real number of each label.
  arXiv  Detail & Related papers  (2021-10-25T12:50:26Z)
• Coping with Label Shift via Distributionally Robust Optimisation [72.80971421083937]
  We propose a model that minimises an objective based on distributionally robust optimisation (DRO) We then design and analyse a gradient descent-proximal mirror ascent algorithm tailored for large-scale problems to optimise the proposed objective.
  arXiv  Detail & Related papers  (2020-10-23T08:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.