Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment

• URL: http://arxiv.org/abs/2603.27987v1
• Date: Mon, 30 Mar 2026 03:20:27 GMT
• Title: Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment
• Abstract summary: We propose a diffusion-based Noise-Optimization (N) method to synthesize a small yet representative set of samples, and "Opt" to augment the synthetic data via "Doping"<n>DsCo is applicable in both data-accessible and data-free scenarios, achieving SOTA performances for low data volumes, and it extends well to high data volumes, where it nearly reduces the dataset size by half with no performance degradation.
• Score: 43.678155518039745
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The high cost and accessibility problem associated with large datasets hinder the development of large-scale visual recognition systems. Dataset Distillation addresses these problems by synthesizing compact surrogate datasets for efficient training, storage, transfer, and privacy preservation. The existing state-of-the-art diffusion-based dataset distillation methods face three issues: lack of theoretical justification, poor efficiency in scaling to high data volumes, and failure in data-free scenarios. To address these issues, we establish a theoretical framework that justifies the use of diffusion models by proving the equivalence between dataset distillation and distribution matching, and reveals an inherent efficiency limit in the dataset distillation paradigm. We then propose a Dataset Concentration (DsCo) framework that uses a diffusion-based Noise-Optimization (NOpt) method to synthesize a small yet representative set of samples, and optionally augments the synthetic data via "Doping", which mixes selected samples from the original dataset with the synthetic samples to overcome the efficiency limit of dataset distillation. DsCo is applicable in both data-accessible and data-free scenarios, achieving SOTA performances for low data volumes, and it extends well to high data volumes, where it nearly reduces the dataset size by half with no performance degradation.

Related papers

• IMS3: Breaking Distributional Aggregation in Diffusion-Based Dataset Distillation [41.83834685276882]
  Recently, diffusion-based dataset distillation methods have shown great promise by leveraging the strong generative capacity of diffusion models to produce diverse and structurally consistent samples.<n>Inversion-Matching (IM) introduces an inversion-guided fine-tuning process that aligns denoising trajectories with their inversion counterparts, broadening distributional coverage and enhancing diversity.<n> Selective Subgroup Sampling(S3) is a training-free sampling mechanism that improves inter-class separability by selecting synthetic subsets that are both representative and distinctive.
  arXiv  Detail & Related papers  (2026-03-14T14:15:03Z)
• State Diversity Matters in Offline Behavior Distillation [58.116300485427764]
  We show that datasets with greater state diversity outperforms those with higher state quality when training loss is substantial.<n>We propose a novel yet simple algorithm, state density weighted (SDW) OBD, which emphasizes state diversity by weighting the distillation objective.
  arXiv  Detail & Related papers  (2025-12-07T06:55:08Z)
• Diffusion Models as Dataset Distillation Priors [39.4727398182562]
  We propose As Diffusion Priors (DAP), which formalizes representativeness by quantifying the similarity between synthetic and real data in feature space.<n>DAP outperforms state-of-the-art methods in generating high-fidelity datasets.<n>Our work establishes a theoretical connection between diffusion priors and the objectives of dataset distillation.
  arXiv  Detail & Related papers  (2025-10-20T11:04:09Z)
• Taming Diffusion for Dataset Distillation with High Representativeness [49.3818035378669]
  D3HR is a novel diffusion-based framework to generate distilled datasets with high representativeness.<n>Our experiments demonstrate that D3HR can achieve higher accuracy across different model architectures.
  arXiv  Detail & Related papers  (2025-05-23T22:05:59Z)
• Diffusion-Augmented Coreset Expansion for Scalable Dataset Distillation [18.474302012851087]
  We propose a two-stage solution for dataset distillation.<n>First, we compress the dataset by selecting only the most informative patches to form a coreset.<n>Next, we leverage a generative foundation model to dynamically expand this compressed set in real-time.<n>We demonstrate a significant improvement of over 10% compared to the state-of-the-art on several large-scale dataset distillation benchmarks.
  arXiv  Detail & Related papers  (2024-12-05T23:40:27Z)
• Importance-Aware Adaptive Dataset Distillation [53.79746115426363]
  Development of deep learning models is enabled by the availability of large-scale datasets. dataset distillation aims to synthesize a compact dataset that retains the essential information from the large original dataset. We propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance.
  arXiv  Detail & Related papers  (2024-01-29T03:29:39Z)
• Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation [96.92250565207017]
  We study the data efficiency and selection for the dataset distillation task. By re-formulating the dynamics of distillation, we provide insight into the inherent redundancy in the real dataset. We find the most contributing samples based on their causal effects on the distillation.
  arXiv  Detail & Related papers  (2023-05-28T06:53:41Z)
• A Comprehensive Survey of Dataset Distillation [73.15482472726555]
  It has become challenging to handle the unlimited growth of data with limited computing power. Deep learning technology has developed unprecedentedly in the last decade. This paper provides a holistic understanding of dataset distillation from multiple aspects.
  arXiv  Detail & Related papers  (2023-01-13T15:11:38Z)
• Dataset Condensation via Efficient Synthetic-Data Parameterization [40.56817483607132]
  Machine learning with massive amounts of data comes at a price of huge computation costs and storage for training and tuning. Recent studies on dataset condensation attempt to reduce the dependence on such massive data by synthesizing a compact training dataset. We propose a novel condensation framework that generates multiple synthetic data with a limited storage budget via efficient parameterization considering data regularity.
  arXiv  Detail & Related papers  (2022-05-30T09:55:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.