Boosting Test Performance with Importance Sampling--a Subpopulation Perspective

• URL: http://arxiv.org/abs/2412.13003v1
• Date: Tue, 17 Dec 2024 15:25:24 GMT
• Title: Boosting Test Performance with Importance Sampling--a Subpopulation Perspective
• Authors: Hongyu Shen, Zhizhen Zhao,
• Abstract summary: In this paper, we identify important sampling as a simple yet powerful tool for solving the subpopulation problem. We provide a new systematic formulation of the subpopulation problem and explicitly identify the assumptions that are not clearly stated in the existing works. On the application side, we demonstrate a single estimator is enough to solve the subpopulation problem.
• Score: 16.678910111353307
• License:
• Abstract: Despite empirical risk minimization (ERM) is widely applied in the machine learning community, its performance is limited on data with spurious correlation or subpopulation that is introduced by hidden attributes. Existing literature proposed techniques to maximize group-balanced or worst-group accuracy when such correlation presents, yet, at the cost of lower average accuracy. In addition, many existing works conduct surveys on different subpopulation methods without revealing the inherent connection between these methods, which could hinder the technology advancement in this area. In this paper, we identify important sampling as a simple yet powerful tool for solving the subpopulation problem. On the theory side, we provide a new systematic formulation of the subpopulation problem and explicitly identify the assumptions that are not clearly stated in the existing works. This helps to uncover the cause of the dropped average accuracy. We provide the first theoretical discussion on the connections of existing methods, revealing the core components that make them different. On the application side, we demonstrate a single estimator is enough to solve the subpopulation problem. In particular, we introduce the estimator in both attribute-known and -unknown scenarios in the subpopulation setup, offering flexibility in practical use cases. And empirically, we achieve state-of-the-art performance on commonly used benchmark datasets.

Related papers

• A step towards the integration of machine learning and small area estimation [0.0]
  We propose a predictor supported by machine learning algorithms which can be used to predict any population or subpopulation characteristics. We study only small departures from the assumed model, to show that our proposal is a good alternative in this case as well. What is more, we propose the method of the accuracy estimation of machine learning predictors, giving the possibility of the accuracy comparison with classic methods.
  arXiv  Detail & Related papers  (2024-02-12T09:43:17Z)
• Optimal Multi-Distribution Learning [88.3008613028333]
  Multi-distribution learning seeks to learn a shared model that minimizes the worst-case risk across $k$ distinct data distributions. We propose a novel algorithm that yields an varepsilon-optimal randomized hypothesis with a sample complexity on the order of (d+k)/varepsilon2.
  arXiv  Detail & Related papers  (2023-12-08T16:06:29Z)
• Multi-dimensional domain generalization with low-rank structures [18.565189720128856]
  In statistical and machine learning methods, it is typically assumed that the test data are identically distributed with the training data. This assumption does not always hold, especially in applications where the target population are not well-represented in the training data. We present a novel approach to addressing this challenge in linear regression models.
  arXiv  Detail & Related papers  (2023-09-18T08:07:58Z)
• Statistical Inference for Fairness Auditing [4.318555434063274]
  We frame this task as "fairness auditing," in terms of multiple hypothesis testing. We show how the bootstrap can be used to simultaneously bound performance disparities over a collection of groups. Our methods can be used to flag subpopulations affected by model underperformance, and certify subpopulations for which the model performs adequately.
  arXiv  Detail & Related papers  (2023-05-05T17:54:22Z)
• Reweighted Mixup for Subpopulation Shift [63.1315456651771]
  Subpopulation shift exists in many real-world applications, which refers to the training and test distributions that contain the same subpopulation groups but with different subpopulation proportions. Importance reweighting is a classical and effective way to handle the subpopulation shift. We propose a simple yet practical framework, called reweighted mixup, to mitigate the overfitting issue.
  arXiv  Detail & Related papers  (2023-04-09T03:44:50Z)
• UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup [44.0372420908258]
  Subpopulation shift wildly exists in many real-world machine learning applications. Importance reweighting is a normal way to handle the subpopulation shift issue. We propose uncertainty-aware mixup (Umix) to mitigate the overfitting issue.
  arXiv  Detail & Related papers  (2022-09-19T11:22:28Z)
• Adaptive Identification of Populations with Treatment Benefit in Clinical Trials: Machine Learning Challenges and Solutions [78.31410227443102]
  We study the problem of adaptively identifying patient subpopulations that benefit from a given treatment during a confirmatory clinical trial. We propose AdaGGI and AdaGCPI, two meta-algorithms for subpopulation construction.
  arXiv  Detail & Related papers  (2022-08-11T14:27:49Z)
• On Modality Bias Recognition and Reduction [70.69194431713825]
  We study the modality bias problem in the context of multi-modal classification. We propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned. Our method yields remarkable performance improvements compared with the baselines.
  arXiv  Detail & Related papers  (2022-02-25T13:47:09Z)
• Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
  Learning from implicit feedback is challenging because of the difficult nature of the one-class problem. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
  arXiv  Detail & Related papers  (2021-05-11T03:38:16Z)
• BREEDS: Benchmarks for Subpopulation Shift [98.90314444545204]
  We develop a methodology for assessing the robustness of models to subpopulation shift. We leverage the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions. Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity.
  arXiv  Detail & Related papers  (2020-08-11T17:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.