FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation

• URL: http://arxiv.org/abs/2506.24125v1
• Date: Mon, 30 Jun 2025 17:59:34 GMT
• Title: FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation
• Authors: Jiacheng Cui, Xinyue Bi, Yaxin Luo, Xiaohan Zhao, Jiacheng Liu, Zhiqiang Shen,
• Abstract summary: Residual connection has been extensively studied and widely applied at the model architecture level.<n>We introduce the concept of Data Residual Matching for the first time, leveraging data-level skip connections to facilitate data generation and mitigate data information vanishing.
• Score: 21.910537847630067
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Residual connection has been extensively studied and widely applied at the model architecture level. However, its potential in the more challenging data-centric approaches remains unexplored. In this work, we introduce the concept of Data Residual Matching for the first time, leveraging data-level skip connections to facilitate data generation and mitigate data information vanishing. This approach maintains a balance between newly acquired knowledge through pixel space optimization and existing core local information identification within raw data modalities, specifically for the dataset distillation task. Furthermore, by incorporating optimization-level refinements, our method significantly improves computational efficiency, achieving superior performance while reducing training time and peak GPU memory usage by 50%. Consequently, the proposed method Fast and Accurate Data Residual Matching for Dataset Distillation (FADRM) establishes a new state-of-the-art, demonstrating substantial improvements over existing methods across multiple dataset benchmarks in both efficiency and effectiveness. For instance, with ResNet-18 as the student model and a 0.8% compression ratio on ImageNet-1K, the method achieves 47.7% test accuracy in single-model dataset distillation and 50.0% in multi-model dataset distillation, surpassing RDED by +5.7% and outperforming state-of-the-art multi-model approaches, EDC and CV-DD, by +1.4% and +4.0%. Code is available at: https://github.com/Jiacheng8/FADRM.

Related papers

• DD-Ranking: Rethinking the Evaluation of Dataset Distillation [223.28392857127733]
  We propose DD-Ranking, a unified evaluation framework, along with new general evaluation metrics to uncover the true performance improvements achieved by different methods.<n>By refocusing on the actual information enhancement of distilled datasets, DD-Ranking provides a more comprehensive and fair evaluation standard for future research advancements.
  arXiv  Detail & Related papers  (2025-05-19T16:19:50Z)
• Dataset Distillation via Committee Voting [21.018818924580877]
  We introduce $bf C$ommittee $bf V$oting for $bf D$ataset $bf D$istillation (CV-DD)<n>CV-DD is a novel approach that leverages the collective wisdom of multiple models or experts to create high-quality distilled datasets.
  arXiv  Detail & Related papers  (2025-01-13T18:59:48Z)
• Generative Dataset Distillation Based on Self-knowledge Distillation [49.20086587208214]
  We present a novel generative dataset distillation method that can improve the accuracy of aligning prediction logits.<n>Our approach integrates self-knowledge distillation to achieve more precise distribution matching between the synthetic and original data.<n>Our method outperforms existing state-of-the-art methods, resulting in superior distillation performance.
  arXiv  Detail & Related papers  (2025-01-08T00:43:31Z)
• Importance-Aware Adaptive Dataset Distillation [53.79746115426363]
  Development of deep learning models is enabled by the availability of large-scale datasets. dataset distillation aims to synthesize a compact dataset that retains the essential information from the large original dataset. We propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance.
  arXiv  Detail & Related papers  (2024-01-29T03:29:39Z)
• Dataset Distillation via Adversarial Prediction Matching [24.487950991247764]
  We propose an adversarial framework to solve the dataset distillation problem efficiently. Our method can produce synthetic datasets just 10% the size of the original, yet achieve, on average, 94% of the test accuracy of models trained on the full original datasets.
  arXiv  Detail & Related papers  (2023-12-14T13:19:33Z)
• On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm [8.220508517570577]
  We propose RDED to enable both diversity and realism of the distilled data. We can distill the full ImageNet-1K to a small dataset comprising 10 images per class within 7 minutes, achieving a notable 42% top-1 accuracy.
  arXiv  Detail & Related papers  (2023-12-06T14:40:05Z)
• Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents [43.282328554697564]
  We propose a highly efficient FL dataset distillation framework on the server side. Unlike previous strategies, our technique enables the server to leverage prior knowledge from pre-trained deep generative models. Our framework converges faster than the baselines because rather than the server trains on several sets of heterogeneous data distributions, it trains on a multi-modal distribution.
  arXiv  Detail & Related papers  (2023-12-03T23:30:48Z)
• Improved Distribution Matching for Dataset Condensation [91.55972945798531]
  We propose a novel dataset condensation method based on distribution matching. Our simple yet effective method outperforms most previous optimization-oriented methods with much fewer computational resources.
  arXiv  Detail & Related papers  (2023-07-19T04:07:33Z)
• Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation [96.92250565207017]
  We study the data efficiency and selection for the dataset distillation task. By re-formulating the dynamics of distillation, we provide insight into the inherent redundancy in the real dataset. We find the most contributing samples based on their causal effects on the distillation.
  arXiv  Detail & Related papers  (2023-05-28T06:53:41Z)
• Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation [151.70234052015948]
  We propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7%.
  arXiv  Detail & Related papers  (2022-11-20T15:49:11Z)
• Dataset Distillation using Neural Feature Regression [32.53291298089172]
  We develop an algorithm for dataset distillation using neural Feature Regression with Pooling (FRePo) FRePo achieves state-of-the-art performance with an order of magnitude less memory requirement and two orders of magnitude faster training than previous methods. We show that high-quality distilled data can greatly improve various downstream applications, such as continual learning and membership inference defense.
  arXiv  Detail & Related papers  (2022-06-01T19:02:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.