Dataset Condensation with Differentiable Siamese Augmentation

• URL: http://arxiv.org/abs/2102.08259v1
• Date: Tue, 16 Feb 2021 16:32:21 GMT
• Title: Dataset Condensation with Differentiable Siamese Augmentation
• Authors: Bo Zhao, Hakan Bilen
• Abstract summary: We focus on condensing large training sets into significantly smaller synthetic sets which can be used to train deep neural networks. We propose Differentiable Siamese Augmentation that enables effective use of data augmentation to synthesize more informative synthetic images. We show with only less than 1% data that our method achieves 99.6%, 94.9%, 88.5%, 71.5% relative performance on MNIST, FashionMNIST, SVHN, CIFAR10 respectively.
• Score: 30.571335208276246
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In many machine learning problems, large-scale datasets have become the de-facto standard to train state-of-the-art deep networks at the price of heavy computation load. In this paper, we focus on condensing large training sets into significantly smaller synthetic sets which can be used to train deep neural networks from scratch with minimum drop in performance. Inspired from the recent training set synthesis methods, we propose Differentiable Siamese Augmentation that enables effective use of data augmentation to synthesize more informative synthetic images and thus achieves better performance when training networks with augmentations. Experiments on multiple image classification benchmarks demonstrate that the proposed method obtains substantial gains over the state-of-the-art, 7% improvements on CIFAR10 and CIFAR100 datasets. We show with only less than 1% data that our method achieves 99.6%, 94.9%, 88.5%, 71.5% relative performance on MNIST, FashionMNIST, SVHN, CIFAR10 respectively.

Related papers

• Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality [78.6359306550245]
  We argue that using just one synthetic subset for distillation will not yield optimal generalization performance. PDD synthesizes multiple small sets of synthetic images, each conditioned on the previous sets, and trains the model on the cumulative union of these subsets. Our experiments show that PDD can effectively improve the performance of existing dataset distillation methods by up to 4.3%.
  arXiv  Detail & Related papers  (2023-10-10T20:04:44Z)
• DataDAM: Efficient Dataset Distillation with Attention Matching [15.300968899043498]
  Researchers have long tried to minimize training costs in deep learning by maintaining strong generalization across diverse datasets. Emerging research on dataset aims to reduce training costs by creating a small synthetic set that contains the information of a larger real dataset. However, the synthetic data generated by previous methods are not guaranteed to distribute and discriminate as well as the original training data.
  arXiv  Detail & Related papers  (2023-09-29T19:07:48Z)
• Improving Resnet-9 Generalization Trained on Small Datasets [4.977981835063451]
  The challenge is to achieve the highest possible accuracy in an image classification task in less than 10 minutes. The training is done on a small dataset of 5000 images picked randomly from CIFAR-10 dataset. Our experiments show that the ResNet-9 can achieve the accuracy of 88% while trained only on a 10% subset of CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2023-09-07T18:46:52Z)
• Dataset Quantization [72.61936019738076]
  We present dataset quantization (DQ), a new framework to compress large-scale datasets into small subsets. DQ is the first method that can successfully distill large-scale datasets such as ImageNet-1k with a state-of-the-art compression ratio.
  arXiv  Detail & Related papers  (2023-08-21T07:24:29Z)
• Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training [110.79400526706081]
  Vision transformers (ViTs) have recently obtained success in many applications, but their intensive computation and heavy memory usage limit their generalization. Previous compression algorithms usually start from the pre-trained dense models and only focus on efficient inference. This paper proposes an end-to-end efficient training framework from three sparse perspectives, dubbed Tri-Level E-ViT.
  arXiv  Detail & Related papers  (2022-11-19T21:15:47Z)
• ScoreMix: A Scalable Augmentation Strategy for Training GANs with Limited Data [93.06336507035486]
  Generative Adversarial Networks (GANs) typically suffer from overfitting when limited training data is available. We present ScoreMix, a novel and scalable data augmentation approach for various image synthesis tasks.
  arXiv  Detail & Related papers  (2022-10-27T02:55:15Z)
• A Study on the Impact of Data Augmentation for Training Convolutional Neural Networks in the Presence of Noisy Labels [14.998309259808236]
  Label noise is common in large real-world datasets, and its presence harms the training process of deep neural networks. We evaluate the impact of data augmentation as a design choice for training deep neural networks. We show that the appropriate selection of data augmentation can drastically improve the model robustness to label noise.
  arXiv  Detail & Related papers  (2022-08-23T20:04:17Z)
• Training Efficient CNNS: Tweaking the Nuts and Bolts of Neural Networks for Lighter, Faster and Robust Models [0.0]
  We demonstrate how an efficient deep convolution network can be built in a phased manner by sequentially reducing the number of training parameters. We achieved a SOTA accuracy of 99.2% on MNIST data with just 1500 parameters and an accuracy of 86.01% with just over 140K parameters on the CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2022-05-23T13:51:06Z)
• Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation [77.34726150561087]
  This article explores various data distillation techniques that can reduce the amount of data required to successfully train deep networks. Inspired by recent ideas, we suggest new data distillation techniques based on generative teaching networks, gradient matching, and the Implicit Function Theorem.
  arXiv  Detail & Related papers  (2022-03-16T11:45:32Z)
• Dataset Condensation with Distribution Matching [30.571335208276246]
  dataset condensation aims to replace the original large training set with a significantly smaller learned synthetic set. We propose a simple yet effective dataset condensation technique that requires significantly lower training cost. Thanks to its efficiency, we apply our method to more realistic and larger datasets with sophisticated neural architectures.
  arXiv  Detail & Related papers  (2021-10-08T15:02:30Z)
• Differentiable Augmentation for Data-Efficient GAN Training [48.920992130257595]
  We propose DiffAugment, a simple method that improves the data efficiency of GANs by imposing various types of differentiable augmentations on both real and fake samples. Our method can generate high-fidelity images using only 100 images without pre-training, while being on par with existing transfer learning algorithms.
  arXiv  Detail & Related papers  (2020-06-18T17:59:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.