Sample Weight Averaging for Stable Prediction

• URL: http://arxiv.org/abs/2502.07414v1
• Date: Tue, 11 Feb 2025 09:51:22 GMT
• Title: Sample Weight Averaging for Stable Prediction
• Authors: Han Yu, Yue He, Renzhe Xu, Dongbai Li, Jiayin Zhang, Wenchao Zou, Peng Cui,
• Abstract summary: The challenge of Out-of-Distribution generalization poses a foundational concern for the application of machine learning algorithms to risk-sensitive areas. We propose SAmple Weight Averaging (SAWA), a simple yet efficacious strategy that can be universally integrated into various sample reweighting algorithms to decrease the variance and coefficient estimation error.
• Score: 33.96006836156484
• License:
• Abstract: The challenge of Out-of-Distribution (OOD) generalization poses a foundational concern for the application of machine learning algorithms to risk-sensitive areas. Inspired by traditional importance weighting and propensity weighting methods, prior approaches employ an independence-based sample reweighting procedure. They aim at decorrelating covariates to counteract the bias introduced by spurious correlations between unstable variables and the outcome, thus enhancing generalization and fulfilling stable prediction under covariate shift. Nonetheless, these methods are prone to experiencing an inflation of variance, primarily attributable to the reduced efficacy in utilizing training samples during the reweighting process. Existing remedies necessitate either environmental labels or substantially higher time costs along with additional assumptions and supervised information. To mitigate this issue, we propose SAmple Weight Averaging (SAWA), a simple yet efficacious strategy that can be universally integrated into various sample reweighting algorithms to decrease the variance and coefficient estimation error, thus boosting the covariate-shift generalization and achieving stable prediction across different environments. We prove its rationality and benefits theoretically. Experiments across synthetic datasets and real-world datasets consistently underscore its superiority against covariate shift.

Related papers

• SeWA: Selective Weight Average via Probabilistic Masking [51.015724517293236]
  We show that only a few points are needed to achieve better and faster convergence. We transform the discrete selection problem into a continuous subset optimization framework. We derive the SeWA's stability bounds, which are sharper than that under both convex image checkpoints.
  arXiv  Detail & Related papers  (2025-02-14T12:35:21Z)
• Heteroscedastic Double Bayesian Elastic Net [1.1240642213359266]
  We propose the Heteroscedastic Double Bayesian Elastic Net (HDBEN), a novel framework that jointly models the mean and log- variance. Our approach simultaneously induces sparsity and grouping in the regression coefficients and variance parameters, capturing complex variance structures in the data.
  arXiv  Detail & Related papers  (2025-02-04T05:44:19Z)
• Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
  We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data. Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
  arXiv  Detail & Related papers  (2024-04-24T09:04:36Z)
• Errors-in-variables Fr\'echet Regression with Low-rank Covariate Approximation [2.1756081703276]
  Fr'echet regression has emerged as a promising approach for regression analysis involving non-Euclidean response variables. Our proposed framework combines the concepts of global Fr'echet regression and principal component regression, aiming to improve the efficiency and accuracy of the regression estimator.
  arXiv  Detail & Related papers  (2023-05-16T08:37:54Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• Stable Learning via Sparse Variable Independence [41.632242102167844]
  We propose SVI (Sparse Variable Independence) for the covariate-shift generalization problem. We introduce sparsity constraint to compensate for the imperfectness of sample reweighting under the finite-sample setting. Experiments on both synthetic and real-world datasets demonstrate the improvement of SVI.
  arXiv  Detail & Related papers  (2022-12-02T05:59:30Z)
• Vector-Valued Least-Squares Regression under Output Regularity Assumptions [73.99064151691597]
  We propose and analyse a reduced-rank method for solving least-squares regression problems with infinite dimensional output. We derive learning bounds for our method, and study under which setting statistical performance is improved in comparison to full-rank method.
  arXiv  Detail & Related papers  (2022-11-16T15:07:00Z)
• Adaptive Importance Sampling meets Mirror Descent: a Bias-variance tradeoff [7.538482310185135]
  A major drawback of adaptive importance sampling is the large variance of the weights. This paper investigates a regularization strategy whose basic principle is to raise the importance weights at a certain power.
  arXiv  Detail & Related papers  (2021-10-29T07:45:24Z)
• A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
  A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution. We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
  arXiv  Detail & Related papers  (2020-07-08T11:35:47Z)
• Balance-Subsampled Stable Prediction [55.13512328954456]
  We propose a novel balance-subsampled stable prediction (BSSP) algorithm based on the theory of fractional factorial design. A design-theoretic analysis shows that the proposed method can reduce the confounding effects among predictors induced by the distribution shift. Numerical experiments on both synthetic and real-world data sets demonstrate that our BSSP algorithm significantly outperforms the baseline methods for stable prediction across unknown test data.
  arXiv  Detail & Related papers  (2020-06-08T07:01:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.