Learning from a Biased Sample
- URL: http://arxiv.org/abs/2209.01754v1
- Date: Mon, 5 Sep 2022 04:19:16 GMT
- Title: Learning from a Biased Sample
- Authors: Roshni Sahoo, Lihua Lei, Stefan Wager
- Abstract summary: We propose a method for learning a decision rule that minimizes the worst-case risk incurred under a family of test distributions.
We give statistical guarantees for learning a robust model using the method of sieves and propose a deep learning algorithm whose loss function captures our target.
- Score: 5.162622771922123
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The empirical risk minimization approach to data-driven decision making
assumes that we can learn a decision rule from training data drawn under the
same conditions as the ones we want to deploy it under. However, in a number of
settings, we may be concerned that our training sample is biased, and that some
groups (characterized by either observable or unobservable attributes) may be
under- or over-represented relative to the general population; and in this
setting empirical risk minimization over the training set may fail to yield
rules that perform well at deployment. Building on concepts from
distributionally robust optimization and sensitivity analysis, we propose a
method for learning a decision rule that minimizes the worst-case risk incurred
under a family of test distributions whose conditional distributions of
outcomes $Y$ given covariates $X$ differ from the conditional training
distribution by at most a constant factor, and whose covariate distributions
are absolutely continuous with respect to the covariate distribution of the
training data. We apply a result of Rockafellar and Uryasev to show that this
problem is equivalent to an augmented convex risk minimization problem. We give
statistical guarantees for learning a robust model using the method of sieves
and propose a deep learning algorithm whose loss function captures our
robustness target. We empirically validate our proposed method in simulations
and a case study with the MIMIC-III dataset.
Related papers
- Active Data Sampling and Generation for Bias Remediation [0.0]
A mixed active sampling and data generation strategy -- called samplation -- is proposed to compensate during fine-tuning of a pre-trained classifer the unfair classifications it produces.
Using as case study Deep Models for visual semantic role labeling, the proposed method has been able to fully cure a simulated gender bias starting from a 90/10 imbalance.
arXiv Detail & Related papers (2025-03-26T10:42:15Z) - Causal Lifting of Neural Representations: Zero-Shot Generalization for Causal Inferences [56.23412698865433]
We focus on causal inferences on a target experiment with unlabeled factual outcomes, retrieved by a predictive model fine-tuned on a labeled similar experiment.
First, we show that factual outcome estimation via Empirical Risk Minimization (ERM) may fail to yield valid causal inferences on the target population.
We propose Deconfounded Empirical Risk Minimization (DERM), a new simple learning procedure minimizing the risk over a fictitious target population.
arXiv Detail & Related papers (2025-02-10T10:52:17Z) - Estimating the Probabilities of Rare Outputs in Language Models [8.585890569162267]
We study low probability estimation in the context of argmax sampling from small transformer language models.
We find that importance sampling outperforms activation extrapolation, but both outperform naive sampling.
We argue that new methods for low probability estimation are needed to provide stronger guarantees about worst-case performance.
arXiv Detail & Related papers (2024-10-17T04:31:18Z) - Rejection via Learning Density Ratios [50.91522897152437]
Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.
We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.
Our framework is tested empirically over clean and noisy datasets.
arXiv Detail & Related papers (2024-05-29T01:32:17Z) - User-defined Event Sampling and Uncertainty Quantification in Diffusion
Models for Physical Dynamical Systems [49.75149094527068]
We show that diffusion models can be adapted to make predictions and provide uncertainty quantification for chaotic dynamical systems.
We develop a probabilistic approximation scheme for the conditional score function which converges to the true distribution as the noise level decreases.
We are able to sample conditionally on nonlinear userdefined events at inference time, and matches data statistics even when sampling from the tails of the distribution.
arXiv Detail & Related papers (2023-06-13T03:42:03Z) - Conformal Inference for Invariant Risk Minimization [12.049545417799125]
The application of machine learning models can be significantly impeded by the occurrence of distributional shifts.
One way to tackle this problem is to use invariant learning, such as invariant risk minimization (IRM), to acquire an invariant representation.
This paper develops methods for obtaining distribution-free prediction regions to describe uncertainty estimates for invariant representations.
arXiv Detail & Related papers (2023-05-22T03:48:38Z) - Unrolling Particles: Unsupervised Learning of Sampling Distributions [102.72972137287728]
Particle filtering is used to compute good nonlinear estimates of complex systems.
We show in simulations that the resulting particle filter yields good estimates in a wide range of scenarios.
arXiv Detail & Related papers (2021-10-06T16:58:34Z) - Unsupervised Embedding Learning from Uncertainty Momentum Modeling [37.674449317054716]
We propose a novel solution to explicitly model and explore the uncertainty of the given unlabeled learning samples.
We leverage such uncertainty modeling momentum to the learning which is helpful to tackle the outliers.
arXiv Detail & Related papers (2021-07-19T14:06:19Z) - Tracking disease outbreaks from sparse data with Bayesian inference [55.82986443159948]
The COVID-19 pandemic provides new motivation for estimating the empirical rate of transmission during an outbreak.
Standard methods struggle to accommodate the partial observability and sparse data common at finer scales.
We propose a Bayesian framework which accommodates partial observability in a principled manner.
arXiv Detail & Related papers (2020-09-12T20:37:33Z) - Distributional Reinforcement Learning via Moment Matching [54.16108052278444]
We formulate a method that learns a finite set of statistics from each return distribution via neural networks.
Our method can be interpreted as implicitly matching all orders of moments between a return distribution and its Bellman target.
Experiments on the suite of Atari games show that our method outperforms the standard distributional RL baselines.
arXiv Detail & Related papers (2020-07-24T05:18:17Z) - A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution.
We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
arXiv Detail & Related papers (2020-07-08T11:35:47Z) - Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning [20.116219345579154]
Decision problems in science, engineering and economics are affected by uncertain parameters whose distribution is only indirectly observable through samples.
The goal of data-driven decision-making is to learn a decision from finitely many training samples that will perform well on unseen test samples.
We will show that Wasserstein distributionally robust optimization has interesting ramifications for statistical learning.
arXiv Detail & Related papers (2019-08-23T09:28:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.