How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization

• URL: http://arxiv.org/abs/2210.06441v2
• Date: Fri, 31 Mar 2023 00:08:46 GMT
• Title: How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization
• Authors: Jonas Geiping, Micah Goldblum, Gowthami Somepalli, Ravid Shwartz-Ziv, Tom Goldstein, Andrew Gordon Wilson
• Abstract summary: We find that in out-of-distribution testing scenarios, augmentations which yield samples that are diverse, but inconsistent with the data distribution can be even more valuable than additional training data. We show that augmentations induce additionality during training, effectively flattening the loss landscape.
• Score: 76.58017437197859
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the clear performance benefits of data augmentations, little is known about why they are so effective. In this paper, we disentangle several key mechanisms through which data augmentations operate. Establishing an exchange rate between augmented and additional real data, we find that in out-of-distribution testing scenarios, augmentations which yield samples that are diverse, but inconsistent with the data distribution can be even more valuable than additional training data. Moreover, we find that data augmentations which encourage invariances can be more valuable than invariance alone, especially on small and medium sized training sets. Following this observation, we show that augmentations induce additional stochasticity during training, effectively flattening the loss landscape.

Related papers

• Revisiting Adversarial Training under Long-Tailed Distributions [8.187045490998269]
  Adversarial training has been recognized as one of the most effective methods to counter such attacks. We show that Balanced Softmax Loss alone can achieve performance comparable to the complete RoBal approach while significantly reducing training overheads. To address this, we explore data augmentation as a solution and unexpectedly discover that, unlike results obtained with balanced data, data augmentation not only effectively alleviates robust overfitting but also significantly improves robustness.
  arXiv  Detail & Related papers  (2024-03-15T07:29:41Z)
• Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updates [3.5253513747455303]
  We identify general aspects of data augmentation (DA) responsible for observed learning improvements. Our study focuses on sparse-reward tasks with dynamics-invariant data augmentation functions.
  arXiv  Detail & Related papers  (2023-10-26T21:28:50Z)
• DualAug: Exploiting Additional Heavy Augmentation with OOD Data Rejection [77.6648187359111]
  We propose a novel data augmentation method, named textbfDualAug, to keep the augmentation in distribution as much as possible at a reasonable time and computational cost. Experiments on supervised image classification benchmarks show that DualAug improve various automated data augmentation method.
  arXiv  Detail & Related papers  (2023-10-12T08:55:10Z)
• On Counterfactual Data Augmentation Under Confounding [30.76982059341284]
  Counterfactual data augmentation has emerged as a method to mitigate confounding biases in the training data. These biases arise due to various observed and unobserved confounding variables in the data generation process. We show how our simple augmentation method helps existing state-of-the-art methods achieve good results.
  arXiv  Detail & Related papers  (2023-05-29T16:20:23Z)
• Automatic Data Augmentation via Invariance-Constrained Learning [94.27081585149836]
  Underlying data structures are often exploited to improve the solution of learning tasks. Data augmentation induces these symmetries during training by applying multiple transformations to the input data. This work tackles these issues by automatically adapting the data augmentation while solving the learning task.
  arXiv  Detail & Related papers  (2022-09-29T18:11:01Z)
• Augmentation-Aware Self-Supervision for Data-Efficient GAN Training [68.81471633374393]
  Training generative adversarial networks (GANs) with limited data is challenging because the discriminator is prone to overfitting. We propose a novel augmentation-aware self-supervised discriminator that predicts the augmentation parameter of the augmented data. We compare our method with state-of-the-art (SOTA) methods using the class-conditional BigGAN and unconditional StyleGAN2 architectures.
  arXiv  Detail & Related papers  (2022-05-31T10:35:55Z)
• Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks [79.13089902898848]
  Self-supervised learning is a powerful paradigm for representation learning on unlabelled images. We show that different tasks in computer vision require features to encode different (in)variances.
  arXiv  Detail & Related papers  (2021-11-22T18:16:35Z)
• Improving Transferability of Representations via Augmentation-Aware Self-Supervision [117.15012005163322]
  AugSelf is an auxiliary self-supervised loss that learns the difference of augmentation parameters between two randomly augmented samples. Our intuition is that AugSelf encourages to preserve augmentation-aware information in learned representations, which could be beneficial for their transferability. AugSelf can easily be incorporated into recent state-of-the-art representation learning methods with a negligible additional training cost.
  arXiv  Detail & Related papers  (2021-11-18T10:43:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.