Learning from a Biased Sample

• URL: http://arxiv.org/abs/2209.01754v3
• Date: Tue, 08 Oct 2024 14:46:11 GMT
• Title: Learning from a Biased Sample
• Authors: Roshni Sahoo, Lihua Lei, Stefan Wager,
• Abstract summary: We propose a method for learning a decision rule that minimizes the worst-case risk incurred under a family of test distributions. We empirically validate our proposed method in a case study on prediction of mental health scores from health survey data.
• Score: 3.546358664345473
• License:
• Abstract: The empirical risk minimization approach to data-driven decision making requires access to training data drawn under the same conditions as those that will be faced when the decision rule is deployed. However, in a number of settings, we may be concerned that our training sample is biased in the sense that some groups (characterized by either observable or unobservable attributes) may be under- or over-represented relative to the general population; and in this setting empirical risk minimization over the training set may fail to yield rules that perform well at deployment. We propose a model of sampling bias called conditional $\Gamma$-biased sampling, where observed covariates can affect the probability of sample selection arbitrarily much but the amount of unexplained variation in the probability of sample selection is bounded by a constant factor. Applying the distributionally robust optimization framework, we propose a method for learning a decision rule that minimizes the worst-case risk incurred under a family of test distributions that can generate the training distribution under $\Gamma$-biased sampling. We apply a result of Rockafellar and Uryasev to show that this problem is equivalent to an augmented convex risk minimization problem. We give statistical guarantees for learning a model that is robust to sampling bias via the method of sieves, and propose a deep learning algorithm whose loss function captures our robust learning target. We empirically validate our proposed method in a case study on prediction of mental health scores from health survey data and a case study on ICU length of stay prediction.

Related papers

• Estimating the Probabilities of Rare Outputs in Language Models [8.585890569162267]
  We study low probability estimation in the context of argmax sampling from small transformer language models. We find that importance sampling outperforms activation extrapolation, but both outperform naive sampling. We argue that new methods for low probability estimation are needed to provide stronger guarantees about worst-case performance.
  arXiv  Detail & Related papers  (2024-10-17T04:31:18Z)
• Distributionally Robust Safe Sample Screening [15.791952053731448]
  We propose a machine learning method called Distributionally Robust Safe Sample Screening (DRSSS) DRSSS aims to identify unnecessary training samples, even when the distribution of the training samples changes in the future. We provide a theoretical guarantee for the DRSSS method and validate its performance through numerical experiments on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-06-10T01:46:42Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions. We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance. Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems [49.75149094527068]
  We show that diffusion models can be adapted to make predictions and provide uncertainty quantification for chaotic dynamical systems. We develop a probabilistic approximation scheme for the conditional score function which converges to the true distribution as the noise level decreases. We are able to sample conditionally on nonlinear userdefined events at inference time, and matches data statistics even when sampling from the tails of the distribution.
  arXiv  Detail & Related papers  (2023-06-13T03:42:03Z)
• Conformal Inference for Invariant Risk Minimization [12.049545417799125]
  The application of machine learning models can be significantly impeded by the occurrence of distributional shifts. One way to tackle this problem is to use invariant learning, such as invariant risk minimization (IRM), to acquire an invariant representation. This paper develops methods for obtaining distribution-free prediction regions to describe uncertainty estimates for invariant representations.
  arXiv  Detail & Related papers  (2023-05-22T03:48:38Z)
• Unrolling Particles: Unsupervised Learning of Sampling Distributions [102.72972137287728]
  Particle filtering is used to compute good nonlinear estimates of complex systems. We show in simulations that the resulting particle filter yields good estimates in a wide range of scenarios.
  arXiv  Detail & Related papers  (2021-10-06T16:58:34Z)
• Unsupervised Embedding Learning from Uncertainty Momentum Modeling [37.674449317054716]
  We propose a novel solution to explicitly model and explore the uncertainty of the given unlabeled learning samples. We leverage such uncertainty modeling momentum to the learning which is helpful to tackle the outliers.
  arXiv  Detail & Related papers  (2021-07-19T14:06:19Z)
• Tracking disease outbreaks from sparse data with Bayesian inference [55.82986443159948]
  The COVID-19 pandemic provides new motivation for estimating the empirical rate of transmission during an outbreak. Standard methods struggle to accommodate the partial observability and sparse data common at finer scales. We propose a Bayesian framework which accommodates partial observability in a principled manner.
  arXiv  Detail & Related papers  (2020-09-12T20:37:33Z)
• Distributional Reinforcement Learning via Moment Matching [54.16108052278444]
  We formulate a method that learns a finite set of statistics from each return distribution via neural networks. Our method can be interpreted as implicitly matching all orders of moments between a return distribution and its Bellman target. Experiments on the suite of Atari games show that our method outperforms the standard distributional RL baselines.
  arXiv  Detail & Related papers  (2020-07-24T05:18:17Z)
• A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
  A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution. We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
  arXiv  Detail & Related papers  (2020-07-08T11:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.