Robust Learning with Progressive Data Expansion Against Spurious Correlation

• URL: http://arxiv.org/abs/2306.04949v2
• Date: Sun, 29 Oct 2023 03:03:32 GMT
• Title: Robust Learning with Progressive Data Expansion Against Spurious Correlation
• Authors: Yihe Deng, Yu Yang, Baharan Mirzasoleiman, Quanquan Gu
• Abstract summary: We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
• Score: 65.83104529677234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While deep learning models have shown remarkable performance in various tasks, they are susceptible to learning non-generalizable spurious features rather than the core features that are genuinely correlated to the true label. In this paper, beyond existing analyses of linear models, we theoretically examine the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. In light of this, we propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance. PDE begins with a group-balanced subset of training data and progressively expands it to facilitate the learning of the core features. Experiments on synthetic and real-world benchmark datasets confirm the superior performance of our method on models such as ResNets and Transformers. On average, our method achieves a 2.8% improvement in worst-group accuracy compared with the state-of-the-art method, while enjoying up to 10x faster training efficiency. Codes are available at https://github.com/uclaml/PDE.

Related papers

• Data Augmentation for Sparse Multidimensional Learning Performance Data Using Generative AI [17.242331892899543]
  Learning performance data describe correct and incorrect answers or problem-solving attempts in adaptive learning. Learning performance data tend to be highly sparse (80%(sim)90% missing observations) in most real-world applications due to adaptive item selection. This article proposes a systematic framework for augmenting learner data to address data sparsity in learning performance data.
  arXiv  Detail & Related papers  (2024-09-24T00:25:07Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• PIGNet2: A Versatile Deep Learning-based Protein-Ligand Interaction Prediction Model for Binding Affinity Scoring and Virtual Screening [0.0]
  Prediction of protein-ligand interactions (PLI) plays a crucial role in drug discovery. The development of a versatile model capable of accurately scoring binding affinity and conducting efficient virtual screening remains a challenge. Here, we propose a viable solution by introducing a novel data augmentation strategy combined with a physics-informed graph neural network.
  arXiv  Detail & Related papers  (2023-07-03T14:46:49Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• Deep Negative Correlation Classification [82.45045814842595]
  Existing deep ensemble methods naively train many different models and then aggregate their predictions. We propose deep negative correlation classification (DNCC) DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
  arXiv  Detail & Related papers  (2022-12-14T07:35:20Z)
• Can we achieve robustness from data alone? [0.7366405857677227]
  Adversarial training and its variants have come to be the prevailing methods to achieve adversarially robust classification using neural networks. We devise a meta-learning method for robust classification, that optimize the dataset prior to its deployment in a principled way. Experiments on MNIST and CIFAR-10 demonstrate that the datasets we produce enjoy very high robustness against PGD attacks.
  arXiv  Detail & Related papers  (2022-07-24T12:14:48Z)
• Training Efficiency and Robustness in Deep Learning [2.6451769337566406]
  We study approaches to improve the training efficiency and robustness of deep learning models. We find that prioritizing learning on more informative training data increases convergence speed and improves generalization performance on test data. We show that a redundancy-aware modification to the sampling of training data improves the training speed and develops an efficient method for detecting the diversity of training signal.
  arXiv  Detail & Related papers  (2021-12-02T17:11:33Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Self-Damaging Contrastive Learning [92.34124578823977]
  Unlabeled data in reality is commonly imbalanced and shows a long-tail distribution. This paper proposes a principled framework called Self-Damaging Contrastive Learning to automatically balance the representation learning without knowing the classes. Our experiments show that SDCLR significantly improves not only overall accuracies but also balancedness.
  arXiv  Detail & Related papers  (2021-06-06T00:04:49Z)
• An Efficient Method of Training Small Models for Regression Problems with Knowledge Distillation [1.433758865948252]
  We propose a new formalism of knowledge distillation for regression problems. First, we propose a new loss function, teacher outlier loss rejection, which rejects outliers in training samples using teacher model predictions. By considering the multi-task network, training of the feature extraction of student models becomes more effective.
  arXiv  Detail & Related papers  (2020-02-28T08:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.