Distilling Datasets Into Less Than One Image

• URL: http://arxiv.org/abs/2403.12040v1
• Date: Mon, 18 Mar 2024 17:59:49 GMT
• Title: Distilling Datasets Into Less Than One Image
• Authors: Asaf Shul, Eliahu Horwitz, Yedid Hoshen,
• Abstract summary: We push the boundaries of dataset distillation, compressing the dataset into less than an image-per-class. Our method establishes a new state-of-the-art performance on CIFAR-10, CIFAR-100, and CUB200 using as little as 0.3 images-per-class.
• Score: 39.08927346274156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dataset distillation aims to compress a dataset into a much smaller one so that a model trained on the distilled dataset achieves high accuracy. Current methods frame this as maximizing the distilled classification accuracy for a budget of K distilled images-per-class, where K is a positive integer. In this paper, we push the boundaries of dataset distillation, compressing the dataset into less than an image-per-class. It is important to realize that the meaningful quantity is not the number of distilled images-per-class but the number of distilled pixels-per-dataset. We therefore, propose Poster Dataset Distillation (PoDD), a new approach that distills the entire original dataset into a single poster. The poster approach motivates new technical solutions for creating training images and learnable labels. Our method can achieve comparable or better performance with less than an image-per-class compared to existing methods that use one image-per-class. Specifically, our method establishes a new state-of-the-art performance on CIFAR-10, CIFAR-100, and CUB200 using as little as 0.3 images-per-class.

Related papers

• Curriculum Coarse-to-Fine Selection for High-IPC Dataset Distillation [67.34754791044242]
  We introduce a novel curriculum coarse-to-fine selection (CCFS) method for efficient high-IPC dataset distillation. CCFS employs a coarse-to-fine strategy to select appropriate real data based on the current synthetic dataset in each curriculum.
  arXiv  Detail & Related papers  (2025-03-24T16:47:40Z)
• Rethinking Large-scale Dataset Compression: Shifting Focus From Labels to Images [60.42768987736088]
  We introduce a benchmark that equitably evaluates methodologies across both distillation and pruning literatures. Our benchmark reveals that in the mainstream dataset distillation setting for large-scale datasets, even randomly selected subsets can achieve surprisingly competitive performance. We propose a new framework for dataset compression, termed Prune, Combine, and Augment (PCA), which focuses on leveraging image data exclusively.
  arXiv  Detail & Related papers  (2025-02-10T13:11:40Z)
• Generative Dataset Distillation Based on Diffusion Model [45.305885410046116]
  We propose a novel generative dataset distillation method based on Stable Diffusion. Specifically, we use the SDXL-Turbo model which can generate images at high speed and quality. We achieved third place in the generative track of the ECCV 2024 DD Challenge.
  arXiv  Detail & Related papers  (2024-08-16T08:52:02Z)
• Low-Rank Similarity Mining for Multimodal Dataset Distillation [50.45577048854653]
  We propose Low-Rank Similarity Mining (LoRS) for multimodal dataset distillation. LoRS distills a ground truth similarity matrix with image-text pairs, and leverages low-rank factorization for efficiency and scalability.
  arXiv  Detail & Related papers  (2024-06-06T07:05:20Z)
• Dataset Distillation via Adversarial Prediction Matching [24.487950991247764]
  We propose an adversarial framework to solve the dataset distillation problem efficiently. Our method can produce synthetic datasets just 10% the size of the original, yet achieve, on average, 94% of the test accuracy of models trained on the full original datasets.
  arXiv  Detail & Related papers  (2023-12-14T13:19:33Z)
• Vision-Language Dataset Distillation [26.886260846439612]
  We design the first vision-language dataset distillation method, building on the idea of trajectory matching. A key challenge is that vision-language datasets do not have a set of discrete classes. Our proposed method jointly distills image-text pairs in a contrastive formulation.
  arXiv  Detail & Related papers  (2023-08-15T03:22:40Z)
• Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation [96.92250565207017]
  We study the data efficiency and selection for the dataset distillation task. By re-formulating the dynamics of distillation, we provide insight into the inherent redundancy in the real dataset. We find the most contributing samples based on their causal effects on the distillation.
  arXiv  Detail & Related papers  (2023-05-28T06:53:41Z)
• Generalizing Dataset Distillation via Deep Generative Prior [75.9031209877651]
  We propose to distill an entire dataset's knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. We present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model's latent space.
  arXiv  Detail & Related papers  (2023-05-02T17:59:31Z)
• DiM: Distilling Dataset into Generative Model [42.32433831074992]
  We propose a novel distillation scheme to textbfDistill information of large train sets textbfinto generative textbfModels, named DiM. During the distillation phase, we minimize the differences in logits predicted by a models pool between real and generated images. At the deployment stage, the generative model synthesizes various training samples from random noises on the fly.
  arXiv  Detail & Related papers  (2023-03-08T16:48:24Z)
• Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory [66.035487142452]
  We show that trajectory-matching-based methods (MTT) can scale to large-scale datasets such as ImageNet-1K. We propose a procedure to exactly compute the unrolled gradient with constant memory complexity, which allows us to scale MTT to ImageNet-1K seamlessly with 6x reduction in memory footprint. The resulting algorithm sets new SOTA on ImageNet-1K: we can scale up to 50 IPCs (Image Per Class) on ImageNet-1K on a single GPU.
  arXiv  Detail & Related papers  (2022-11-19T04:46:03Z)
• Large-Scale Generative Data-Free Distillation [17.510996270055184]
  We propose a new method to train a generative image model by leveraging the intrinsic normalization layers' statistics. The proposed method pushes forward the data-free distillation performance on CIFAR-10 and CIFAR-100 to 95.02% and 77.02% respectively. We are able to scale it to ImageNet dataset, which to the best of our knowledge, has never been done using generative models in a data-free setting.
  arXiv  Detail & Related papers  (2020-12-10T10:54:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.