Data-Efficient Augmentation for Training Neural Networks

• URL: http://arxiv.org/abs/2210.08363v3
• Date: Thu, 20 Jul 2023 05:41:18 GMT
• Title: Data-Efficient Augmentation for Training Neural Networks
• Authors: Tian Yu Liu and Baharan Mirzasoleiman
• Abstract summary: We propose a rigorous technique to select subsets of data points that when augmented, closely capture the training dynamics of full data augmentation. Our method achieves 6.3x speedup on CIFAR10 and 2.2x speedup on SVHN, and outperforms the baselines by up to 10% across various subset sizes.
• Score: 15.870155099135538
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data augmentation is essential to achieve state-of-the-art performance in many deep learning applications. However, the most effective augmentation techniques become computationally prohibitive for even medium-sized datasets. To address this, we propose a rigorous technique to select subsets of data points that when augmented, closely capture the training dynamics of full data augmentation. We first show that data augmentation, modeled as additive perturbations, improves learning and generalization by relatively enlarging and perturbing the smaller singular values of the network Jacobian, while preserving its prominent directions. This prevents overfitting and enhances learning the harder to learn information. Then, we propose a framework to iteratively extract small subsets of training data that when augmented, closely capture the alignment of the fully augmented Jacobian with labels/residuals. We prove that stochastic gradient descent applied to the augmented subsets found by our approach has similar training dynamics to that of fully augmented data. Our experiments demonstrate that our method achieves 6.3x speedup on CIFAR10 and 2.2x speedup on SVHN, and outperforms the baselines by up to 10% across various subset sizes. Similarly, on TinyImageNet and ImageNet, our method beats the baselines by up to 8%, while achieving up to 3.3x speedup across various subset sizes. Finally, training on and augmenting 50% subsets using our method on a version of CIFAR10 corrupted with label noise even outperforms using the full dataset. Our code is available at: https://github.com/tianyu139/data-efficient-augmentation

Related papers

• Replication: Contrastive Learning and Data Augmentation in Traffic Classification Using a Flowpic Input Representation [47.95762911696397]
  We reproduce [16] on the same datasets and replicate its most salient aspect (the importance of data augmentation) on three additional public datasets. While we confirm most of the original results, we also found a 20% accuracy drop on some of the investigated scenarios due to a data shift in the original dataset.
  arXiv  Detail & Related papers  (2023-09-18T12:55:09Z)
• Dynamic Data Augmentation via MCTS for Prostate MRI Segmentation [19.780410411548935]
  We present Dynamic Data Augmentation (DDAug), which is efficient and has negligible cost. DDAug computation develops a hierarchical tree structure to represent various augmentations. Our method outperforms the current state-of-the-art data augmentation strategies.
  arXiv  Detail & Related papers  (2023-05-25T06:44:43Z)
• GraphLearner: Graph Node Clustering with Fully Learnable Augmentation [76.63963385662426]
  Contrastive deep graph clustering (CDGC) leverages the power of contrastive learning to group nodes into different clusters. We propose a Graph Node Clustering with Fully Learnable Augmentation, termed GraphLearner. It introduces learnable augmentors to generate high-quality and task-specific augmented samples for CDGC.
  arXiv  Detail & Related papers  (2022-12-07T10:19:39Z)
• Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training [110.79400526706081]
  Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage limit their generalization. Previous compression algorithms usually start from the pre-trained dense models and only focus on efficient inference. This paper proposes an end-to-end efficient training framework from three sparse perspectives, dubbed Tri-Level E-ViT.
  arXiv  Detail & Related papers  (2022-11-19T21:15:47Z)
• Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations [76.82124752950148]
  We develop a convenient gradient-based method for selecting the data augmentation. We use a differentiable Kronecker-factored Laplace approximation to the marginal likelihood as our objective.
  arXiv  Detail & Related papers  (2022-02-22T02:51:11Z)
• Feature transforms for image data augmentation [74.12025519234153]
  In image classification, many augmentation approaches utilize simple image manipulation algorithms. In this work, we build ensembles on the data level by adding images generated by combining fourteen augmentation approaches. Pretrained ResNet50 networks are finetuned on training sets that include images derived from each augmentation method.
  arXiv  Detail & Related papers  (2022-01-24T14:12:29Z)
• Dynamic Data Augmentation with Gating Networks [5.251019642214251]
  We propose a neural network that dynamically selects the best combination using a mutually beneficial gating network and a feature consistency loss. In experiments, we demonstrate the effectiveness of the proposed method on the 12 largest time-series datasets from 2018 UCR Time Series Archive.
  arXiv  Detail & Related papers  (2021-11-05T04:24:51Z)
• Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding [69.40915115518523]
  Lack of training data presents a grand challenge to scaling out spoken language understanding (SLU) to low-resource languages. Various data augmentation approaches have been proposed to synthesize training data in low-resource target languages. In this paper we focus on mitigating noise in augmented data.
  arXiv  Detail & Related papers  (2021-09-03T15:44:15Z)
• Reducing Labelled Data Requirement for Pneumonia Segmentation using Image Augmentations [0.0]
  We investigate the effect of image augmentations on reducing the requirement of labelled data in semantic segmentation of chest X-rays for pneumonia detection. We train fully convolutional network models on subsets of different sizes from the total training data. We find that rotate and mixup are the best augmentations amongst rotate, mixup, translate, gamma and horizontal flip, wherein they reduce the labelled data requirement by 70%.
  arXiv  Detail & Related papers  (2021-02-25T10:11:30Z)
• Regularizing Deep Networks with Semantic Data Augmentation [44.53483945155832]
  We propose a novel semantic data augmentation algorithm to complement traditional approaches. The proposed method is inspired by the intriguing property that deep networks are effective in learning linearized features. We show that the proposed implicit semantic data augmentation (ISDA) algorithm amounts to minimizing a novel robust CE loss.
  arXiv  Detail & Related papers  (2020-07-21T00:32:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.