Importance Weight Estimation and Generalization in Domain Adaptation under Label Shift

• URL: http://arxiv.org/abs/2011.14251v1
• Date: Sun, 29 Nov 2020 01:37:58 GMT
• Title: Importance Weight Estimation and Generalization in Domain Adaptation under Label Shift
• Authors: Kamyar Azizzadenesheli
• Abstract summary: We study generalization under label shift in domain adaptation where the learner has access to labeled samples from the source domain but unlabeled samples from the target domain. We introduce a new operator learning approach between Hilbert spaces defined on labels. We show the generalization property of the importance weighted empirical risk minimization on the unseen target domain.
• Score: 8.10196482629998
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study generalization under label shift in domain adaptation where the learner has access to labeled samples from the source domain but unlabeled samples from the target domain. Prior works deploy label classifiers and introduce various methods to estimate the importance weights from source to target domains. They use these estimates in importance weighted empirical risk minimization to learn classifiers. In this work, we theoretically compare the prior approaches, relax their strong assumptions, and generalize them from requiring label classifiers to general functions. This latter generalization improves the conditioning on the inverse operator of the induced inverse problems by allowing for broader exploitation of the spectrum of the forward operator. The prior works in the study of label shifts are limited to categorical label spaces. In this work, we propose a series of methods to estimate the importance weight functions for arbitrary normed label spaces. We introduce a new operator learning approach between Hilbert spaces defined on labels (rather than covariates) and show that it induces a perturbed inverse problem of compact operators. We propose a novel approach to solve the inverse problem in the presence of perturbation. This analysis has its own independent interest since such problems commonly arise in partial differential equations and reinforcement learning. For both categorical and general normed spaces, we provide concentration bounds for the proposed estimators. Using the existing generalization analysis based on Rademacher complexity, R\'enyi divergence, and MDFR lemma in Azizzadenesheli et al. [2019], we show the generalization property of the importance weighted empirical risk minimization on the unseen target domain.

Related papers

• Guidance Not Obstruction: A Conjugate Consistent Enhanced Strategy for Domain Generalization [50.04665252665413]
  We argue that acquiring discriminative generalization between classes within domains is crucial. In contrast to seeking distribution alignment, we endeavor to safeguard domain-related between-class discrimination. We employ a novel distribution-level Universum strategy to generate supplementary diverse domain-related class-conditional distributions.
  arXiv  Detail & Related papers  (2024-12-13T12:25:16Z)
• Generalization bounds for regression and classification on adaptive covering input domains [1.4141453107129398]
  We focus on the generalization bound, which serves as an upper limit for the generalization error. In the case of classification tasks, we treat the target function as a one-hot, a piece-wise constant function, and employ 0/1 loss for error measurement.
  arXiv  Detail & Related papers  (2024-07-29T05:40:08Z)
• Probabilistic Test-Time Generalization by Variational Neighbor-Labeling [62.158807685159736]
  This paper strives for domain generalization, where models are trained exclusively on source domains before being deployed on unseen target domains. Probability pseudo-labeling of target samples to generalize the source-trained model to the target domain at test time. Variational neighbor labels that incorporate the information of neighboring target samples to generate more robust pseudo labels.
  arXiv  Detail & Related papers  (2023-07-08T18:58:08Z)
• Quantitatively Measuring and Contrastively Exploring Heterogeneity for Domain Generalization [38.50749918578154]
  We propose Heterogeneity-based Two-stage Contrastive Learning (HTCL) for the Domain generalization task. In the first stage, we generate the most heterogeneous dividing pattern with our contrastive metric. In the second stage, we employ an in-aimed contrastive learning by re-building pairs with the stable relation hinted by domains and classes.
  arXiv  Detail & Related papers  (2023-05-25T09:42:43Z)
• Label Alignment Regularization for Distribution Shift [63.228879525056904]
  Recent work has highlighted the label alignment property (LAP) in supervised learning, where the vector of all labels in the dataset is mostly in the span of the top few singular vectors of the data matrix. We propose a regularization method for unsupervised domain adaptation that encourages alignment between the predictions in the target domain and its top singular vectors. We report improved performance over domain adaptation baselines in well-known tasks such as MNIST-USPS domain adaptation and cross-lingual sentiment analysis.
  arXiv  Detail & Related papers  (2022-11-27T22:54:48Z)
• ER: Equivariance Regularizer for Knowledge Graph Completion [107.51609402963072]
  We propose a new regularizer, namely, Equivariance Regularizer (ER) ER can enhance the generalization ability of the model by employing the semantic equivariance between the head and tail entities. The experimental results indicate a clear and substantial improvement over the state-of-the-art relation prediction methods.
  arXiv  Detail & Related papers  (2022-06-24T08:18:05Z)
• Fundamental Limits and Tradeoffs in Invariant Representation Learning [99.2368462915979]
  Many machine learning applications involve learning representations that achieve two competing goals. Minimax game-theoretic formulation represents a fundamental tradeoff between accuracy and invariance. We provide an information-theoretic analysis of this general and important problem under both classification and regression settings.
  arXiv  Detail & Related papers  (2020-12-19T15:24:04Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)
• Unsupervised Domain Adaptation via Discriminative Manifold Propagation [26.23123292060868]
  Unsupervised domain adaptation is effective in leveraging rich information from a labeled source domain to an unlabeled target domain. The proposed method can be used to tackle a series of variants of domain adaptation problems, including both vanilla and partial settings.
  arXiv  Detail & Related papers  (2020-08-23T12:31:37Z)
• Self-training Avoids Using Spurious Features Under Domain Shift [54.794607791641745]
  In unsupervised domain adaptation, conditional entropy minimization and pseudo-labeling work even when the domain shifts are much larger than those analyzed by existing theory. We identify and analyze one particular setting where the domain shift can be large, but certain spurious features correlate with label in the source domain but are independent label in the target.
  arXiv  Detail & Related papers  (2020-06-17T17:51:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.