Latent Dataset Distillation with Diffusion Models

• URL: http://arxiv.org/abs/2403.03881v3
• Date: Thu, 11 Jul 2024 09:10:10 GMT
• Title: Latent Dataset Distillation with Diffusion Models
• Authors: Brian B. Moser, Federico Raue, Sebastian Palacio, Stanislav Frolov, Andreas Dengel,
• Abstract summary: This paper proposes Latent dataset Distillation with Diffusion Models (LD3M) Our novel diffusion process is tailored for this task and significantly improves the gradient flow for distillation. Overall, LD3M consistently outperforms state-of-the-art methods by up to 4.8 p.p. and 4.2 p.p. for 1 and 10 images per class, respectively.
• Score: 9.398135472047132
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning traditionally relies on increasingly larger datasets. Yet, such datasets pose major storage challenges and usually contain non-influential samples, which could be ignored during training without negatively impacting the training quality. In response, the idea of distilling a dataset into a condensed set of synthetic samples, i.e., a distilled dataset, emerged. One key aspect is the selected architecture, usually ConvNet, for linking the original and synthetic datasets. However, the final accuracy is lower if the employed model architecture differs from that used during distillation. Another challenge is the generation of high-resolution images (128x128 and higher). To address both challenges, this paper proposes Latent Dataset Distillation with Diffusion Models (LD3M) that combine diffusion in latent space with dataset distillation. Our novel diffusion process is tailored for this task and significantly improves the gradient flow for distillation. By adjusting the number of diffusion steps, LD3M also offers a convenient way of controlling the trade-off between distillation speed and dataset quality. Overall, LD3M consistently outperforms state-of-the-art methods by up to 4.8 p.p. and 4.2 p.p. for 1 and 10 images per class, respectively, and on several ImageNet subsets and high resolutions (128x128 and 256x256).

Related papers

• Data-to-Model Distillation: Data-Efficient Learning Framework [14.44010988811002]
  We propose a novel framework called Data-to-Model Distillation (D2M) to distill the real dataset's knowledge into the learnable parameters of a pre-trained generative model. Our method effectively scales up to high-resolution 128x128 ImageNet-1K.
  arXiv  Detail & Related papers  (2024-11-19T20:10:28Z)
• One Step Diffusion-based Super-Resolution with Time-Aware Distillation [60.262651082672235]
  Diffusion-based image super-resolution (SR) methods have shown promise in reconstructing high-resolution images with fine details from low-resolution counterparts. Recent techniques have been devised to enhance the sampling efficiency of diffusion-based SR models via knowledge distillation. We propose a time-aware diffusion distillation method, named TAD-SR, to accomplish effective and efficient image super-resolution.
  arXiv  Detail & Related papers  (2024-08-14T11:47:22Z)
• One Category One Prompt: Dataset Distillation using Diffusion Models [22.512552596310176]
  We introduce Diffusion Models (D3M) as a novel paradigm for dataset distillation, leveraging recent advancements in generative text-to-image foundation models. Our approach utilizes textual inversion, a technique for fine-tuning text-to-image generative models, to create concise and informative representations for large datasets.
  arXiv  Detail & Related papers  (2024-03-11T20:23:59Z)
• Importance-Aware Adaptive Dataset Distillation [53.79746115426363]
  Development of deep learning models is enabled by the availability of large-scale datasets. dataset distillation aims to synthesize a compact dataset that retains the essential information from the large original dataset. We propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance.
  arXiv  Detail & Related papers  (2024-01-29T03:29:39Z)
• Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality [78.6359306550245]
  We argue that using just one synthetic subset for distillation will not yield optimal generalization performance. PDD synthesizes multiple small sets of synthetic images, each conditioned on the previous sets, and trains the model on the cumulative union of these subsets. Our experiments show that PDD can effectively improve the performance of existing dataset distillation methods by up to 4.3%.
  arXiv  Detail & Related papers  (2023-10-10T20:04:44Z)
• Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation [96.92250565207017]
  We study the data efficiency and selection for the dataset distillation task. By re-formulating the dynamics of distillation, we provide insight into the inherent redundancy in the real dataset. We find the most contributing samples based on their causal effects on the distillation.
  arXiv  Detail & Related papers  (2023-05-28T06:53:41Z)
• Generalizing Dataset Distillation via Deep Generative Prior [75.9031209877651]
  We propose to distill an entire dataset's knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. We present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model's latent space.
  arXiv  Detail & Related papers  (2023-05-02T17:59:31Z)
• DiM: Distilling Dataset into Generative Model [42.32433831074992]
  We propose a novel distillation scheme to textbfDistill information of large train sets textbfinto generative textbfModels, named DiM. During the distillation phase, we minimize the differences in logits predicted by a models pool between real and generated images. At the deployment stage, the generative model synthesizes various training samples from random noises on the fly.
  arXiv  Detail & Related papers  (2023-03-08T16:48:24Z)
• Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation [151.70234052015948]
  We propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7%.
  arXiv  Detail & Related papers  (2022-11-20T15:49:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.