Dataset Distillation by Influence Matching

• URL: http://arxiv.org/abs/2607.16859v1
• Date: Sat, 18 Jul 2026 15:47:03 GMT
• Title: Dataset Distillation by Influence Matching
• Abstract summary: Inf-Match learns a compact synthetic set whose effect on the converged parameters matches that of the full dataset.<n>It delivers the best accuracy across standard classification benchmarks.<n>Inf-Match scales to vision-language distillation on Flickr30K, outperforming strong process-matching baselines.
• Score: 42.09931906541133
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We revisit dataset distillation from an outcome-centric perspective. Rather than aligning process surrogates (per-step gradients or training trajectories), Influence Matching (Inf-Match) aligns the final outcome of training: it learns a compact synthetic set whose effect on the converged parameters matches that of the full dataset. Concretely, we introduce a fully differentiable, sample-level influence estimator that quantifies parameter shifts from adding or removing data, without time-consuming inverse-Hessian products or convexity assumptions. The estimator runs in linear time by unrolling the optimization dynamics and applying a first-order Taylor approximation. We then learn the synthetic set by minimizing the mismatch between its influence and that of the real dataset, yielding outcome alignment rather than heuristic process imitation. Inf-Match delivers the best accuracy across standard classification benchmarks. For instance, on Tiny-ImageNet (IPC=10), Inf-Match attains 31.5\%, a +4.7\% improvement over NCFM. Beyond classification, Inf-Match scales to vision-language distillation on Flickr30K, outperforming strong process-matching baselines. For instance, with 200 to 1000 synthetic samples, our method achieved a leading impressive average on image/text retrieval tasks, higher than NCFM by 2.5\%. The code will be released via https://github.com/hrtan/infmatch.

Related papers

• Faithful and Fast Influence Function via Advanced Sampling [10.773767694482858]
  We propose two advanced sampling techniques based on features and logits.<n>These samplers select a small yet representative subset of the entire dataset by considering the distribution of features or logits.<n>We validate our approach through class removal experiments, using the F1-score to measure how effectively the model forgets a class.
  arXiv  Detail & Related papers  (2025-10-30T17:55:19Z)
• Understanding Data Influence with Differential Approximation [63.817689230826595]
  We introduce a new formulation to approximate a sample's influence by accumulating the differences in influence between consecutive learning steps, which we term Diff-In.<n>By employing second-order approximations, we approximate these difference terms with high accuracy while eliminating the need for model convexity required by existing methods.<n>Our theoretical analysis demonstrates that Diff-In achieves significantly lower approximation error compared to existing influence estimators.
  arXiv  Detail & Related papers  (2025-08-20T11:59:32Z)
• Efficient Data Selection at Scale via Influence Distillation [53.03573620682107]
  This paper introduces Influence Distillation, a mathematicallyjustified framework for data selection.<n>By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data.<n>Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to $3.5times$ faster selection.
  arXiv  Detail & Related papers  (2025-05-25T09:08:00Z)
• Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining [17.402771370806384]
  Filter Like You Test (FLYT) is an algorithm for curating datasets that learns the usefulness of each data point as a pretraining example.<n>FLYT trains a scoring model that learns to weigh each example's features using gradient signals from downstream tasks training sets.<n>We implement Mixing-FLYT (M-FLYT), which takes the per-example scores generated by different scoring methods as features, and learns to unify them into a single score.
  arXiv  Detail & Related papers  (2025-03-11T18:34:12Z)
• Dataset Distillation with Neural Characteristic Function: A Minmax Perspective [39.77640775591437]
  We reformulate dataset distillation as a minmax optimization problem and introduce Neural Characteristic Function Discrepancy (NCFD)<n>NCFD is a comprehensive and theoretically grounded metric for measuring distributional differences.<n>Our method achieves significant performance gains over state-of-the-art methods on both low- and high-resolution datasets.
  arXiv  Detail & Related papers  (2025-02-28T02:14:55Z)
• Embarassingly Simple Dataset Distillation [0.0]
  We tackle dataset distillation at its core by treating it directly as a bilevel optimization problem. A deeper dive into the nature of distilled data unveils pronounced intercorrelation. We devise a boosting mechanism that generates distilled datasets that contain subsets with near optimal performance across different data budgets.
  arXiv  Detail & Related papers  (2023-11-13T02:14:54Z)
• Better May Not Be Fairer: A Study on Subgroup Discrepancy in Image Classification [73.87160347728314]
  We investigate how natural background colors play a role as spurious features by annotating the test sets of CIFAR10 and CIFAR100 into subgroups based on the background color of each image. We find that overall human-level accuracy does not guarantee consistent subgroup performances, and the phenomenon remains even on models pre-trained on ImageNet or after data augmentation (DA) Experimental results show that FlowAug achieves more consistent subgroup results than other types of DA methods on CIFAR10/100 and on CIFAR10/100-C.
  arXiv  Detail & Related papers  (2022-12-16T18:51:10Z)
• Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation [151.70234052015948]
  We propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7%.
  arXiv  Detail & Related papers  (2022-11-20T15:49:11Z)
• ScoreMix: A Scalable Augmentation Strategy for Training GANs with Limited Data [93.06336507035486]
  Generative Adversarial Networks (GANs) typically suffer from overfitting when limited training data is available. We present ScoreMix, a novel and scalable data augmentation approach for various image synthesis tasks.
  arXiv  Detail & Related papers  (2022-10-27T02:55:15Z)
• Augmentation Strategies for Learning with Noisy Labels [3.698228929379249]
  We evaluate different augmentation strategies for algorithms tackling the "learning with noisy labels" problem. We find that using one set of augmentations for loss modeling tasks and another set for learning is the most effective. We introduce this augmentation strategy to the state-of-the-art technique and demonstrate that we can improve performance across all evaluated noise levels.
  arXiv  Detail & Related papers  (2021-03-03T02:19:35Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.