Condensing Large-Scale Datasets Directly with Minimal Information Loss

• URL: http://arxiv.org/abs/2607.00916v1
• Date: Wed, 01 Jul 2026 13:21:44 GMT
• Title: Condensing Large-Scale Datasets Directly with Minimal Information Loss
• Abstract summary: CIM is a novel, metric-driven framework that abandons the flawed dual-compression paradigm.<n> CIM explicitly quantifies and minimizes the information gap between the original and synthetic datasets.
• Score: 34.43216741538152
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in scaling dataset distillation rely heavily on decoupled information extraction pipelines, comprising SQUEEZE, RECOVER, and RELABEL stages. Despite their scalability to large-scale datasets, these methods suffer from prohibitive computational overhead and poor cross-architecture generalization. In this paper, we reveal the root cause of these bottlenecks: the implicit dual-compression process, from data to model and back to images, inherently induces severe information loss. Crucially, we empirically and theoretically demonstrate that this loss creates a distribution shift that fundamentally compromises the widely adopted RELABEL strategy, transforming the pre-trained model into an unreliable labeler that yields sub-optimal labels. To overcome these critical flaws, we propose CIM, a novel, metric-driven framework that abandons the flawed dual-compression paradigm. Instead, CIM explicitly quantifies and minimizes the information gap between the original and synthetic datasets. By directly aligning the data distributions, our approach ensures high-fidelity information condensation and inherently satisfies the prerequisites for effective relabeling. Extensive experiments demonstrate that CIM establishes a new state-of-the-art. Notably, it distills ImageNet-1K at an IPC=10 in merely 80 minutes on a single RTX-4090 GPU, achieving an unprecedented 48.7% Top-1 accuracy on ResNet-18 and significantly outperforming previous SOTA approaches, such as NRR-DD and DELT, by 2.6% and 2.9%, respectively. Our code is available at https://github.com/LINs-lab/CIM.

Related papers

• Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration [71.77541419885925]
  We present a unified graph-based DP framework.<n>By modeling the dataset as a weighted graph, DP can be cast as a Maximum Weight Clique Problem (MWCP)<n>Our method outperforms existing DP methods while substantially reducing training cost, reducing training time by over 40% without sacrificing accuracy on ImageNet-1k with ResNet-50.
  arXiv  Detail & Related papers  (2026-06-11T05:13:32Z)
• CoDA: From Text-to-Image Diffusion Models to Training-Free Dataset Distillation [71.52209438343928]
  Core Distribution Alignment (CoDA) is a framework that enables effective Distillation (DD) using only an off-the-shelf text-to-image model.<n>Our key idea is to first identify the "intrinsic core distribution" of the target dataset using a robust density-based discovery mechanism.<n>By doing so, CoDA effectively bridges the gap between general-purpose generative priors and target semantics.
  arXiv  Detail & Related papers  (2025-12-03T14:45:57Z)
• Maximizing Efficiency of Dataset Compression for Machine Learning Potentials With Information Theory [0.45880283710344066]
  Machine learning interatomic potentials (MLIPs) balance high accuracy and lower costs compared to density functional theory calculations.<n>Large datasets improve model accuracy and generalization but are computationally expensive to produce and train on.<n>We develop an information-theoretical framework to quantify the efficiency of dataset compression methods.
  arXiv  Detail & Related papers  (2025-11-13T18:00:21Z)
• FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation [21.910537847630067]
  Residual connection has been extensively studied and widely applied at the model architecture level.<n>We introduce the concept of Data Residual Matching for the first time, leveraging data-level skip connections to facilitate data generation and mitigate data information vanishing.
  arXiv  Detail & Related papers  (2025-06-30T17:59:34Z)
• Dataset Distillation as Pushforward Optimal Quantization [2.5892916589735457]
  We propose a synthetic training set that achieves similar performance to training on real data, with orders of magnitude less computational requirements.<n>In particular, we link existing disentangled dataset distillation methods to the classical optimal quantization and Wasserstein barycenter problems.<n>We achieve better performance and inter-model generalization on the ImageNet-1K dataset with trivial additional computation, and SOTA performance in higher image-per-class settings.
  arXiv  Detail & Related papers  (2025-01-13T20:41:52Z)
• Self-supervised Dataset Distillation: A Good Compression Is All You Need [23.02066055996762]
  We introduce SC-DD, a simple yet effective Self-supervised Compression framework for dataset distillation. The proposed SC-DD outperforms all previous state-of-the-art supervised dataset distillation methods when employing larger models. Experiments are conducted on CIFAR-100, Tiny-ImageNet and ImageNet-1K datasets to demonstrate the superiority of our proposed approach.
  arXiv  Detail & Related papers  (2024-04-11T17:56:40Z)
• Dataset Distillation via Adversarial Prediction Matching [24.487950991247764]
  We propose an adversarial framework to solve the dataset distillation problem efficiently. Our method can produce synthetic datasets just 10% the size of the original, yet achieve, on average, 94% of the test accuracy of models trained on the full original datasets.
  arXiv  Detail & Related papers  (2023-12-14T13:19:33Z)
• Dataset Quantization [72.61936019738076]
  We present dataset quantization (DQ), a new framework to compress large-scale datasets into small subsets. DQ is the first method that can successfully distill large-scale datasets such as ImageNet-1k with a state-of-the-art compression ratio.
  arXiv  Detail & Related papers  (2023-08-21T07:24:29Z)
• Gradient-Free Structured Pruning with Unlabeled Data [57.999191898036706]
  We propose a gradient-free structured pruning framework that uses only unlabeled data. Up to 40% of the original FLOP count can be reduced with less than a 4% accuracy loss across all tasks considered.
  arXiv  Detail & Related papers  (2023-03-07T19:12:31Z)
• Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation [151.70234052015948]
  We propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7%.
  arXiv  Detail & Related papers  (2022-11-20T15:49:11Z)
• Contrastive Model Inversion for Data-Free Knowledge Distillation [60.08025054715192]
  We propose Contrastive Model Inversion, where the data diversity is explicitly modeled as an optimizable objective. Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI achieves significantly superior performance when the generated data are used for knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-18T15:13:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.