Evaluating Robustness to Dataset Shift via Parametric Robustness Sets
- URL: http://arxiv.org/abs/2205.15947v1
- Date: Tue, 31 May 2022 16:44:18 GMT
- Title: Evaluating Robustness to Dataset Shift via Parametric Robustness Sets
- Authors: Nikolaj Thams, Michael Oberst, David Sontag
- Abstract summary: We give a method for proactively identifying small, plausible shifts in distribution which lead to large differences in model performance.
We apply an approach to classifying gender from images, revealing sensitivity to shifts in non-causal attributes.
- Score: 7.347989843033034
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We give a method for proactively identifying small, plausible shifts in
distribution which lead to large differences in model performance. To ensure
that these shifts are plausible, we parameterize them in terms of interpretable
changes in causal mechanisms of observed variables. This defines a parametric
robustness set of plausible distributions and a corresponding worst-case loss.
While the loss under an individual parametric shift can be estimated via
reweighting techniques such as importance sampling, the resulting worst-case
optimization problem is non-convex, and the estimate may suffer from large
variance. For small shifts, however, we can construct a local second-order
approximation to the loss under shift and cast the problem of finding a
worst-case shift as a particular non-convex quadratic optimization problem, for
which efficient algorithms are available. We demonstrate that this second-order
approximation can be estimated directly for shifts in conditional exponential
family models, and we bound the approximation error. We apply our approach to a
computer vision task (classifying gender from images), revealing sensitivity to
shifts in non-causal attributes.
Related papers
- Decision-Focused Evaluation of Worst-Case Distribution Shift [18.98504221245623]
We introduce a novel framework to identify worst-case distribution shifts in predictive resource allocation settings.
We show that the problem can be reformulated as a submodular optimization problem, enabling efficient approximations of worst-case loss.
Applying our framework to real data, we find empirical evidence that worst-case shifts identified by one metric often significantly diverge from worst-case distributions identified by other metrics.
arXiv Detail & Related papers (2024-07-04T01:00:53Z) - Adapting to Latent Subgroup Shifts via Concepts and Proxies [82.01141290360562]
We show that the optimal target predictor can be non-parametrically identified with the help of concept and proxy variables available only in the source domain.
For continuous observations, we propose a latent variable model specific to the data generation process at hand.
arXiv Detail & Related papers (2022-12-21T18:30:22Z) - On the Pitfalls of Heteroscedastic Uncertainty Estimation with
Probabilistic Neural Networks [23.502721524477444]
We present a synthetic example illustrating how this approach can lead to very poor but stable estimates.
We identify the culprit to be the log-likelihood loss, along with certain conditions that exacerbate the issue.
We present an alternative formulation, termed $beta$-NLL, in which each data point's contribution to the loss is weighted by the $beta$-exponentiated variance estimate.
arXiv Detail & Related papers (2022-03-17T08:46:17Z) - Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic
Uncertainty [58.144520501201995]
Bi-Lipschitz regularization of neural network layers preserve relative distances between data instances in the feature spaces of each layer.
With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices.
We also propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution.
arXiv Detail & Related papers (2021-10-12T22:04:19Z) - Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates.
We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters.
We show that our method improves both accuracy and uncertainty estimation.
arXiv Detail & Related papers (2021-09-27T01:09:08Z) - Predicting with Confidence on Unseen Distributions [90.68414180153897]
We connect domain adaptation and predictive uncertainty literature to predict model accuracy on challenging unseen distributions.
We find that the difference of confidences (DoC) of a classifier's predictions successfully estimates the classifier's performance change over a variety of shifts.
We specifically investigate the distinction between synthetic and natural distribution shifts and observe that despite its simplicity DoC consistently outperforms other quantifications of distributional difference.
arXiv Detail & Related papers (2021-07-07T15:50:18Z) - Robust Bayesian Inference for Discrete Outcomes with the Total Variation
Distance [5.139874302398955]
Models of discrete-valued outcomes are easily misspecified if the data exhibit zero-inflation, overdispersion or contamination.
Here, we introduce a robust discrepancy-based Bayesian approach using the Total Variation Distance (TVD)
We empirically demonstrate that our approach is robust and significantly improves predictive performance on a range of simulated and real world data.
arXiv Detail & Related papers (2020-10-26T09:53:06Z) - Robust, Accurate Stochastic Optimization for Variational Inference [68.83746081733464]
We show that common optimization methods lead to poor variational approximations if the problem is moderately large.
Motivated by these findings, we develop a more robust and accurate optimization framework by viewing the underlying algorithm as producing a Markov chain.
arXiv Detail & Related papers (2020-09-01T19:12:11Z) - Distributionally Robust Bayesian Optimization [121.71766171427433]
We present a novel distributionally robust Bayesian optimization algorithm (DRBO) for zeroth-order, noisy optimization.
Our algorithm provably obtains sub-linear robust regret in various settings.
We demonstrate the robust performance of our method on both synthetic and real-world benchmarks.
arXiv Detail & Related papers (2020-02-20T22:04:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.