Distill Gold from Massive Ores: Efficient Dataset Distillation via
Critical Samples Selection
- URL: http://arxiv.org/abs/2305.18381v3
- Date: Wed, 29 Nov 2023 10:46:19 GMT
- Title: Distill Gold from Massive Ores: Efficient Dataset Distillation via
Critical Samples Selection
- Authors: Yue Xu, Yong-Lu Li, Kaitong Cui, Ziyu Wang, Cewu Lu, Yu-Wing Tai,
Chi-Keung Tang
- Abstract summary: We model the dataset distillation task within the context of information transport.
We introduce and validate a family of data utility estimators and optimal data selection methods to exploit the most valuable samples.
Our method consistently enhances the distillation algorithms, even on much larger-scale and more heterogeneous datasets.
- Score: 101.78275454476311
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Data-efficient learning has garnered significant attention, especially given
the current trend of large multi-modal models. Recently, dataset distillation
becomes an effective approach for data-efficiency; however, the distillation
process itself can still be inefficient. In this work, we model the dataset
distillation task within the context of information transport. By observing the
substantial data redundancy inherent in the distillation, we argue to put more
emphasis on the samples' utility for the distillation task. We introduce and
validate a family of data utility estimators and optimal data selection methods
to exploit the most valuable samples. This strategy significantly reduces the
training costs and extends various existing distillation algorithms to larger
and more diversified datasets, e.g., in some cases only 0.04% training data is
sufficient for comparable distillation performance. Our method consistently
enhances the distillation algorithms, even on much larger-scale and more
heterogeneous datasets, e.g. ImageNet-1K and Kinetics-400. This paradigm opens
up new avenues in the dynamics of distillation and paves the way for efficient
dataset distillation. Our code is available on
https://github.com/silicx/GoldFromOres .
Related papers
- Mitigating Bias in Dataset Distillation [62.79454960378792]
We study the impact of bias inside the original dataset on the performance of dataset distillation.
We introduce a simple yet highly effective approach based on a sample reweighting scheme utilizing kernel density estimation.
arXiv Detail & Related papers (2024-06-06T18:52:28Z) - Exploring the potential of prototype-based soft-labels data distillation for imbalanced data classification [0.0]
Main goal is to push further the performance of prototype-based soft-labels distillation in terms of classification accuracy.
Experimental studies trace the capability of the method to distill the data, but also the opportunity to act as an augmentation method.
arXiv Detail & Related papers (2024-03-25T19:15:19Z) - Importance-Aware Adaptive Dataset Distillation [53.79746115426363]
Development of deep learning models is enabled by the availability of large-scale datasets.
dataset distillation aims to synthesize a compact dataset that retains the essential information from the large original dataset.
We propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance.
arXiv Detail & Related papers (2024-01-29T03:29:39Z) - Dataset Distillation via Adversarial Prediction Matching [24.487950991247764]
We propose an adversarial framework to solve the dataset distillation problem efficiently.
Our method can produce synthetic datasets just 10% the size of the original, yet achieve, on average, 94% of the test accuracy of models trained on the full original datasets.
arXiv Detail & Related papers (2023-12-14T13:19:33Z) - Data Distillation Can Be Like Vodka: Distilling More Times For Better
Quality [78.6359306550245]
We argue that using just one synthetic subset for distillation will not yield optimal generalization performance.
PDD synthesizes multiple small sets of synthetic images, each conditioned on the previous sets, and trains the model on the cumulative union of these subsets.
Our experiments show that PDD can effectively improve the performance of existing dataset distillation methods by up to 4.3%.
arXiv Detail & Related papers (2023-10-10T20:04:44Z) - Explicit and Implicit Knowledge Distillation via Unlabeled Data [5.702176304876537]
We propose an efficient unlabeled sample selection method to replace high computational generators.
We also propose a class-dropping mechanism to suppress the label noise caused by the data domain shifts.
Experimental results show that our method can quickly converge and obtain higher accuracy than other state-of-the-art methods.
arXiv Detail & Related papers (2023-02-17T09:10:41Z) - A Comprehensive Survey of Dataset Distillation [73.15482472726555]
It has become challenging to handle the unlimited growth of data with limited computing power.
Deep learning technology has developed unprecedentedly in the last decade.
This paper provides a holistic understanding of dataset distillation from multiple aspects.
arXiv Detail & Related papers (2023-01-13T15:11:38Z) - Dataset Distillation with Infinitely Wide Convolutional Networks [18.837952916998947]
We apply distributed kernel based meta-learning framework to achieve state-of-the-art results for dataset distillation.
We obtain over 64% test accuracy on CIFAR-10 image classification task, a dramatic improvement over the previous best test accuracy of 40%.
Our state-of-the-art results extend across many other settings for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN.
arXiv Detail & Related papers (2021-07-27T18:31:42Z) - New Properties of the Data Distillation Method When Working With Tabular
Data [77.34726150561087]
Data distillation is the problem of reducing the volume oftraining data while keeping only the necessary information.
We show that the model trained on distilled samples can outperform the model trained on the original dataset.
arXiv Detail & Related papers (2020-10-19T20:27:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.