D$^4$M: Dataset Distillation via Disentangled Diffusion Model

• URL: http://arxiv.org/abs/2407.15138v1
• Date: Sun, 21 Jul 2024 12:16:20 GMT
• Title: D$^4$M: Dataset Distillation via Disentangled Diffusion Model
• Authors: Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, Bowen Tang,
• Abstract summary: We propose an efficient framework for dataset distillation via Disentangled Diffusion Model (D$4$M) Compared to architecture-dependent methods, D$4$M employs latent diffusion model to guarantee consistency and incorporates label information into category prototypes. D$4$M demonstrates superior performance and robust generalization, surpassing the SOTA methods across most aspects.
• Score: 4.568710926635445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dataset distillation offers a lightweight synthetic dataset for fast network training with promising test accuracy. To imitate the performance of the original dataset, most approaches employ bi-level optimization and the distillation space relies on the matching architecture. Nevertheless, these approaches either suffer significant computational costs on large-scale datasets or experience performance decline on cross-architectures. We advocate for designing an economical dataset distillation framework that is independent of the matching architectures. With empirical observations, we argue that constraining the consistency of the real and synthetic image spaces will enhance the cross-architecture generalization. Motivated by this, we introduce Dataset Distillation via Disentangled Diffusion Model (D$^4$M), an efficient framework for dataset distillation. Compared to architecture-dependent methods, D$^4$M employs latent diffusion model to guarantee consistency and incorporates label information into category prototypes. The distilled datasets are versatile, eliminating the need for repeated generation of distinct datasets for various architectures. Through comprehensive experiments, D$^4$M demonstrates superior performance and robust generalization, surpassing the SOTA methods across most aspects.

Related papers

• Data-to-Model Distillation: Data-Efficient Learning Framework [14.44010988811002]
  We propose a novel framework called Data-to-Model Distillation (D2M) to distill the real dataset's knowledge into the learnable parameters of a pre-trained generative model. Our method effectively scales up to high-resolution 128x128 ImageNet-1K.
  arXiv  Detail & Related papers  (2024-11-19T20:10:28Z)
• MDM: Advancing Multi-Domain Distribution Matching for Automatic Modulation Recognition Dataset Synthesis [35.07663680944459]
  Deep learning technology has been successfully introduced into Automatic Modulation Recognition (AMR) tasks. The success of deep learning is all attributed to the training on large-scale datasets. In order to solve the problem of large amount of data, some researchers put forward the method of data distillation.
  arXiv  Detail & Related papers  (2024-08-05T14:16:54Z)
• Hierarchical Features Matter: A Deep Exploration of GAN Priors for Improved Dataset Distillation [51.44054828384487]
  We propose a novel parameterization method dubbed Hierarchical Generative Latent Distillation (H-GLaD) This method systematically explores hierarchical layers within the generative adversarial networks (GANs) In addition, we introduce a novel class-relevant feature distance metric to alleviate the computational burden associated with synthetic dataset evaluation.
  arXiv  Detail & Related papers  (2024-06-09T09:15:54Z)
• ATOM: Attention Mixer for Efficient Dataset Distillation [17.370852204228253]
  We propose a module to efficiently distill large datasets using a mixture of channel and spatial-wise attention. By integrating both types of attention, our ATOM module demonstrates superior performance across various computer vision datasets.
  arXiv  Detail & Related papers  (2024-05-02T15:15:01Z)
• Distribution-Aware Data Expansion with Diffusion Models [55.979857976023695]
  We propose DistDiff, a training-free data expansion framework based on the distribution-aware diffusion model. DistDiff consistently enhances accuracy across a diverse range of datasets compared to models trained solely on original data.
  arXiv  Detail & Related papers  (2024-03-11T14:07:53Z)
• Importance-Aware Adaptive Dataset Distillation [53.79746115426363]
  Development of deep learning models is enabled by the availability of large-scale datasets. dataset distillation aims to synthesize a compact dataset that retains the essential information from the large original dataset. We propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance.
  arXiv  Detail & Related papers  (2024-01-29T03:29:39Z)
• Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study [52.83643622795387]
  Cross-architecture generalization of dataset distillation weakens its practical significance. We propose a novel method of EvaLuation with distillation Feature (ELF) By performing extensive experiments, we successfully prove that ELF can well enhance the cross-architecture generalization of current DD methods.
  arXiv  Detail & Related papers  (2023-12-09T15:41:42Z)
• Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents [43.282328554697564]
  We propose a highly efficient FL dataset distillation framework on the server side. Unlike previous strategies, our technique enables the server to leverage prior knowledge from pre-trained deep generative models. Our framework converges faster than the baselines because rather than the server trains on several sets of heterogeneous data distributions, it trains on a multi-modal distribution.
  arXiv  Detail & Related papers  (2023-12-03T23:30:48Z)
• Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation [151.70234052015948]
  We propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7%.
  arXiv  Detail & Related papers  (2022-11-20T15:49:11Z)
• CAFE: Learning to Condense Dataset by Aligning Features [72.99394941348757]
  We propose a novel scheme to Condense dataset by Aligning FEatures (CAFE) At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-03-03T05:58:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.