Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation

• URL: http://arxiv.org/abs/2412.04668v1
• Date: Thu, 05 Dec 2024 23:40:27 GMT
• Title: Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation
• Authors: Ali Abbasi, Shima Imani, Chenyang An, Gayathri Mahalingam, Harsh Shrivastava, Maurice Diesendruck, Hamed Pirsiavash, Pramod Sharma, Soheil Kolouri,
• Abstract summary: We propose a two-stage solution for dataset distillation. First, we compress the dataset by selecting only the most informative patches to form a coreset. Next, we leverage a generative foundation model to dynamically expand this compressed set in real-time. We demonstrate a significant improvement of over 10% compared to the state-of-the-art on several large-scale dataset distillation benchmarks.
• Score: 18.474302012851087
• License:
• Abstract: With the rapid scaling of neural networks, data storage and communication demands have intensified. Dataset distillation has emerged as a promising solution, condensing information from extensive datasets into a compact set of synthetic samples by solving a bilevel optimization problem. However, current methods face challenges in computational efficiency, particularly with high-resolution data and complex architectures. Recently, knowledge-distillation-based dataset condensation approaches have made this process more computationally feasible. Yet, with the recent developments of generative foundation models, there is now an opportunity to achieve even greater compression, enhance the quality of distilled data, and introduce valuable diversity into the data representation. In this work, we propose a two-stage solution. First, we compress the dataset by selecting only the most informative patches to form a coreset. Next, we leverage a generative foundation model to dynamically expand this compressed set in real-time, enhancing the resolution of these patches and introducing controlled variability to the coreset. Our extensive experiments demonstrate the robustness and efficiency of our approach across a range of dataset distillation benchmarks. We demonstrate a significant improvement of over 10% compared to the state-of-the-art on several large-scale dataset distillation benchmarks. The code will be released soon.

Related papers

• Dataset Distillation as Pushforward Optimal Quantization [1.039189397779466]
  We propose a simple extension of the state-of-the-art data distillation method D4M, achieving better performance on the ImageNet-1K dataset with trivial additional computation. We demonstrate that when equipped with an encoder-decoder structure, the empirically successful disentangled methods can be reformulated as an optimal quantization problem. In particular, we link existing disentangled dataset distillation methods to the classical optimal quantization and Wasserstein barycenter problems, demonstrating consistency of distilled datasets for diffusion-based generative priors.
  arXiv  Detail & Related papers  (2025-01-13T20:41:52Z)
• Adaptive Dataset Quantization [2.0105434963031463]
  We introduce a versatile framework for dataset compression, namely Adaptive dataset Quantization (ADQ) We propose a novel adaptive sampling strategy through the evaluation of generated bins' representativeness score, diversity score and importance score. Our method not only exhibits superior generalization capability across different architectures, but also attains state-of-the-art results, surpassing DQ by average 3% on various datasets.
  arXiv  Detail & Related papers  (2024-12-22T07:08:29Z)
• One Category One Prompt: Dataset Distillation using Diffusion Models [22.512552596310176]
  We introduce Diffusion Models (D3M) as a novel paradigm for dataset distillation, leveraging recent advancements in generative text-to-image foundation models. Our approach utilizes textual inversion, a technique for fine-tuning text-to-image generative models, to create concise and informative representations for large datasets.
  arXiv  Detail & Related papers  (2024-03-11T20:23:59Z)
• Distribution-Aware Data Expansion with Diffusion Models [55.979857976023695]
  We propose DistDiff, a training-free data expansion framework based on the distribution-aware diffusion model. DistDiff consistently enhances accuracy across a diverse range of datasets compared to models trained solely on original data.
  arXiv  Detail & Related papers  (2024-03-11T14:07:53Z)
• Improved Distribution Matching for Dataset Condensation [91.55972945798531]
  We propose a novel dataset condensation method based on distribution matching. Our simple yet effective method outperforms most previous optimization-oriented methods with much fewer computational resources.
  arXiv  Detail & Related papers  (2023-07-19T04:07:33Z)
• Generalizing Dataset Distillation via Deep Generative Prior [75.9031209877651]
  We propose to distill an entire dataset's knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. We present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model's latent space.
  arXiv  Detail & Related papers  (2023-05-02T17:59:31Z)
• A Comprehensive Survey of Dataset Distillation [73.15482472726555]
  It has become challenging to handle the unlimited growth of data with limited computing power. Deep learning technology has developed unprecedentedly in the last decade. This paper provides a holistic understanding of dataset distillation from multiple aspects.
  arXiv  Detail & Related papers  (2023-01-13T15:11:38Z)
• Dataset Distillation via Factorization [58.8114016318593]
  We introduce a emphdataset factorization approach, termed emphHaBa, which is a plug-and-play strategy portable to any existing dataset distillation (DD) baseline. emphHaBa explores decomposing a dataset into two components: data emphHallucination networks and emphBases. Our method can yield significant improvement on downstream classification tasks compared with previous state of the arts, while reducing the total number of compressed parameters by up to 65%.
  arXiv  Detail & Related papers  (2022-10-30T08:36:19Z)
• DC-BENCH: Dataset Condensation Benchmark [79.18718490863908]
  This work provides the first large-scale standardized benchmark on dataset condensation. It consists of a suite of evaluations to comprehensively reflect the generability and effectiveness of condensation methods. The benchmark library is open-sourced to facilitate future research and application.
  arXiv  Detail & Related papers  (2022-07-20T03:54:05Z)
• Dataset Distillation using Neural Feature Regression [32.53291298089172]
  We develop an algorithm for dataset distillation using neural Feature Regression with Pooling (FRePo) FRePo achieves state-of-the-art performance with an order of magnitude less memory requirement and two orders of magnitude faster training than previous methods. We show that high-quality distilled data can greatly improve various downstream applications, such as continual learning and membership inference defense.
  arXiv  Detail & Related papers  (2022-06-01T19:02:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.