Probable Domain Generalization via Quantile Risk Minimization
- URL: http://arxiv.org/abs/2207.09944v4
- Date: Tue, 22 Aug 2023 09:31:35 GMT
- Title: Probable Domain Generalization via Quantile Risk Minimization
- Authors: Cian Eastwood, Alexander Robey, Shashank Singh, Julius von K\"ugelgen,
Hamed Hassani, George J. Pappas, Bernhard Sch\"olkopf
- Abstract summary: Domain generalization seeks predictors which perform well on unseen test distributions.
We propose a new probabilistic framework for DG where the goal is to learn predictors that perform well with high probability.
- Score: 90.15831047587302
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Domain generalization (DG) seeks predictors which perform well on unseen test
distributions by leveraging data drawn from multiple related training
distributions or domains. To achieve this, DG is commonly formulated as an
average- or worst-case problem over the set of possible domains. However,
predictors that perform well on average lack robustness while predictors that
perform well in the worst case tend to be overly-conservative. To address this,
we propose a new probabilistic framework for DG where the goal is to learn
predictors that perform well with high probability. Our key idea is that
distribution shifts seen during training should inform us of probable shifts at
test time, which we realize by explicitly relating training and test domains as
draws from the same underlying meta-distribution. To achieve probable DG, we
propose a new optimization problem called Quantile Risk Minimization (QRM). By
minimizing the $\alpha$-quantile of predictor's risk distribution over domains,
QRM seeks predictors that perform well with probability $\alpha$. To solve QRM
in practice, we propose the Empirical QRM (EQRM) algorithm and provide: (i) a
generalization bound for EQRM; and (ii) the conditions under which EQRM
recovers the causal predictor as $\alpha \to 1$. In our experiments, we
introduce a more holistic quantile-focused evaluation protocol for DG and
demonstrate that EQRM outperforms state-of-the-art baselines on datasets from
WILDS and DomainBed.
Related papers
- Domain Agnostic Conditional Invariant Predictions for Domain Generalization [20.964740750976667]
We propose a Discriminant Risk Minimization (DRM) theory and the corresponding algorithm to capture the invariant features without domain labels.
In DRM theory, we prove that reducing the discrepancy of prediction distribution between overall source domain and any subset of it can contribute to obtaining invariant features.
We evaluate our algorithm against various domain generalization methods on multiple real-world datasets.
arXiv Detail & Related papers (2024-06-09T02:38:52Z) - Whom to Trust? Elective Learning for Distributed Gaussian Process
Regression [3.5208783730894972]
We develop an elective learning algorithm, namely prior-aware elective distributed GP (Pri-GP)
Pri-GP empowers agents with the capability to selectively request predictions from neighboring agents based on their trustworthiness.
We establish a prediction error bound within the Pri-GP framework, ensuring the reliability of predictions.
arXiv Detail & Related papers (2024-02-05T13:52:56Z) - Outage Performance and Novel Loss Function for an ML-Assisted Resource
Allocation: An Exact Analytical Framework [2.1397655110395752]
We introduce a novel loss function to minimize the outage probability of an ML-based resource allocation system.
An ML binary classification predictor assists in selecting a resource satisfying the established outage criterium.
arXiv Detail & Related papers (2023-05-16T18:23:52Z) - Source-free Unsupervised Domain Adaptation for Blind Image Quality
Assessment [20.28784839680503]
Existing learning-based methods for blind image quality assessment (BIQA) are heavily dependent on large amounts of annotated training data.
In this paper, we take the first step towards the source-free unsupervised domain adaptation (SFUDA) in a simple yet efficient manner.
We present a group of well-designed self-supervised objectives to guide the adaptation of the BN affine parameters towards the target domain.
arXiv Detail & Related papers (2022-07-17T09:42:36Z) - Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient
for Out-of-Distribution Generalization [52.7137956951533]
We argue that devising simpler methods for learning predictors on existing features is a promising direction for future research.
We introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift.
Under a natural model, we prove that the DARE solution is the minimax-optimal predictor for a constrained set of test distributions.
arXiv Detail & Related papers (2022-02-14T16:42:16Z) - Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions.
In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data.
We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
arXiv Detail & Related papers (2022-01-11T23:01:12Z) - Iterative Feature Matching: Toward Provable Domain Generalization with
Logarithmic Environments [55.24895403089543]
Domain generalization aims at performing well on unseen test environments with data from a limited number of training environments.
We present a new algorithm based on performing iterative feature matching that is guaranteed with high probability to yield a predictor that generalizes after seeing only $O(logd_s)$ environments.
arXiv Detail & Related papers (2021-06-18T04:39:19Z) - KL Guided Domain Adaptation [88.19298405363452]
Domain adaptation is an important problem and often needed for real-world applications.
A common approach in the domain adaptation literature is to learn a representation of the input that has the same distributions over the source and the target domain.
We show that with a probabilistic representation network, the KL term can be estimated efficiently via minibatch samples.
arXiv Detail & Related papers (2021-06-14T22:24:23Z) - The Risks of Invariant Risk Minimization [52.7137956951533]
Invariant Risk Minimization is an objective based on the idea for learning deep, invariant features of data.
We present the first analysis of classification under the IRM objective--as well as these recently proposed alternatives--under a fairly natural and general model.
We show that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution--this is precisely the issue that it was intended to solve.
arXiv Detail & Related papers (2020-10-12T14:54:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.