Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria

• URL: http://arxiv.org/abs/2212.02457v3
• Date: Mon, 20 May 2024 01:10:48 GMT
• Title: Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria
• Authors: Tengyuan Liang,
• Abstract summary: Covariate distribution shifts and adversarial perturbations present challenges to the conventional statistical learning framework. This paper precisely characterizes the extrapolation region, examining both regression and classification in an infinite-dimensional setting.
• Score: 6.738946307589742
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Covariate distribution shifts and adversarial perturbations present robustness challenges to the conventional statistical learning framework: mild shifts in the test covariate distribution can significantly affect the performance of the statistical model learned based on the training distribution. The model performance typically deteriorates when extrapolation happens: namely, covariates shift to a region where the training distribution is scarce, and naturally, the learned model has little information. For robustness and regularization considerations, adversarial perturbation techniques are proposed as a remedy; however, careful study needs to be carried out about what extrapolation region adversarial covariate shift will focus on, given a learned model. This paper precisely characterizes the extrapolation region, examining both regression and classification in an infinite-dimensional setting. We study the implications of adversarial covariate shifts to subsequent learning of the equilibrium -- the Bayes optimal model -- in a sequential game framework. We exploit the dynamics of the adversarial learning game and reveal the curious effects of the covariate shift to equilibrium learning and experimental design. In particular, we establish two directional convergence results that exhibit distinctive phenomena: (1) a blessing in regression, the adversarial covariate shifts in an exponential rate to an optimal experimental design for rapid subsequent learning; (2) a curse in classification, the adversarial covariate shifts in a subquadratic rate to the hardest experimental design trapping subsequent learning.

Related papers

• Counterfactual Generative Modeling with Variational Causal Inference [1.9287470458589586]
  We present a novel variational Bayesian causal inference framework to handle counterfactual generative modeling tasks. In experiments, we demonstrate the advantage of our framework compared to state-of-the-art models in counterfactual generative modeling.
  arXiv  Detail & Related papers  (2024-10-16T16:44:12Z)
• TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression [11.040033344386366]
  We propose a two-step method with a novel fused-regularizer to improve the learning performance on a target task with limited samples. Nonasymptotic bound is provided for the estimation error of the target model. We extend the method to a distributed setting, allowing for a pretraining-finetuning strategy.
  arXiv  Detail & Related papers  (2024-04-01T14:58:16Z)
• Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchor [18.696420390977863]
  This paper systematically analyzes the forgetting degree of each class during local training across different communication rounds. Motivated by these findings, we propose a novel and straightforward algorithm called Federated Knowledge Anchor (FedKA)
  arXiv  Detail & Related papers  (2023-12-05T01:12:56Z)
• Adapting to Continuous Covariate Shift via Online Density Ratio Estimation [64.8027122329609]
  Dealing with distribution shifts is one of the central challenges for modern machine learning. We propose an online method that can appropriately reuse historical information. Our density ratio estimation method is proven to perform well by enjoying a dynamic regret bound.
  arXiv  Detail & Related papers  (2023-02-06T04:03:33Z)
• Enhancing Adversarial Training with Feature Separability [52.39305978984573]
  We introduce a new concept of adversarial training graph (ATG) with which the proposed adversarial training with feature separability (ATFS) enables to boost the intra-class feature similarity and increase inter-class feature variance. Through comprehensive experiments, we demonstrate that the proposed ATFS framework significantly improves both clean and robust performance.
  arXiv  Detail & Related papers  (2022-05-02T04:04:23Z)
• Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective [72.55093886515824]
  We introduce a causal formalism of motion forecasting, which casts the problem as a dynamic process with three groups of latent variables. We devise a modular architecture that factorizes the representations of invariant mechanisms and style confounders to approximate a causal graph. Experiment results on synthetic and real datasets show that our three proposed components significantly improve the robustness and reusability of the learned motion representations.
  arXiv  Detail & Related papers  (2021-11-29T18:59:09Z)
• Covariate Shift in High-Dimensional Random Feature Regression [44.13449065077103]
  Covariate shift is a significant obstacle in the development of robust machine learning models. We present a theoretical understanding in context of modern machine learning.
  arXiv  Detail & Related papers  (2021-11-16T05:23:28Z)
• Stratified Learning: A General-Purpose Statistical Method for Improved Learning under Covariate Shift [1.1470070927586016]
  We propose a simple, statistically principled, and theoretically justified method to improve supervised learning when the training set is not representative. We build upon a well-established methodology in causal inference, and show that the effects of covariate shift can be reduced or eliminated by conditioning on propensity scores. We demonstrate the effectiveness of our general-purpose method on two contemporary research questions in cosmology, outperforming state-of-the-art importance weighting methods.
  arXiv  Detail & Related papers  (2021-06-21T15:53:20Z)
• Adversarial Robustness through the Lens of Causality [105.51753064807014]
  adversarial vulnerability of deep neural networks has attracted significant attention in machine learning. We propose to incorporate causality into mitigating adversarial vulnerability. Our method can be seen as the first attempt to leverage causality for mitigating adversarial vulnerability.
  arXiv  Detail & Related papers  (2021-06-11T06:55:02Z)
• Double Robust Representation Learning for Counterfactual Prediction [68.78210173955001]
  We propose a novel scalable method to learn double-robust representations for counterfactual predictions. We make robust and efficient counterfactual predictions for both individual and average treatment effects. The algorithm shows competitive performance with the state-of-the-art on real world and synthetic data.
  arXiv  Detail & Related papers  (2020-10-15T16:39:26Z)
• A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
  A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution. We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
  arXiv  Detail & Related papers  (2020-07-08T11:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.