Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models

• URL: http://arxiv.org/abs/2605.07194v1
• Date: Fri, 08 May 2026 03:41:51 GMT
• Title: Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models
• Abstract summary: We propose a closed-form formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver.<n>On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost.<n>On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly $14times$ faster and using less than one-eighth of the GPU memory.
• Score: 37.194177406897964
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dataset distillation compresses a large training set into a small synthetic set that preserves downstream training utility. While most existing methods target training networks from scratch, modern visual transfer learning often uses frozen pre-trained encoders followed by lightweight linear probing. Existing distillation methods for this setting either unroll iterative linear-probe updates with trajectory-based gradient matching, or rely on closed-form formulations originally designed for from-scratch training with neural-tangent-kernel (NTK) approximations. Neither route exploits the fact that frozen-feature linear probing admits a closed-form solution determined directly by the pre-trained features themselves, with no infinite-width approximation and no inner-loop trajectory. We propose Closed-Form Linear-Probe Dataset Distillation (CLP-DD), a bilevel formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver. The synthetic images are then updated by evaluating this induced classifier on real features through a temperature-scaled softmax cross-entropy, where the classifier columns act as learned class anchors in feature space. We further show that the choice of outer objective is decisive: pairing the closed-form inner solver with a standard MSE outer loss substantially underperforms trajectory-based methods, while the discriminative outer loss closes most of the gap. On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost. On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly $14\times$ faster and using less than one-eighth of the GPU memory.

Related papers

• Rethinking Pixel Mean Flows via Interval Denoiser [95.11499828543344]
  We propose the Interval Denoiser, a framework for latent-free generation.<n>Our model achieves an FID of 4.55 in one step (1-NFE) and 3.98 in two steps (2-NFE) without perceptual losses.
  arXiv  Detail & Related papers  (2026-08-05T13:22:43Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Hybrid Iterative Neural Low-Regularity Integrator for Nonlinear Dispersive Equations [0.0]
  HIN-LRI is a hybrid framework that augments a classical numerical solver with a neural operator trained to correct the solver's structured truncation error.<n>A lightweight neural network, operating on a low-dimensional latent manifold, learns the residual defect that analytical methods cannot close.<n>Experiments show that HIN-LRI improves accuracy over analytical, splitting methods, and neural PDE surrogates, with stable spatial refinement, effective out-of-distribution transfer, and modest online overhead.
  arXiv  Detail & Related papers  (2026-05-06T12:50:36Z)
• ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation [9.230247128710865]
  We propose a training-free diffusion-based framework that integrates manifold consistent guidance at every denoising timestep.<n>ManifoldGD improves representativeness, diversity, and image fidelity without requiring any model retraining.
  arXiv  Detail & Related papers  (2026-02-26T18:07:10Z)
• Parallel Diffusion Solver via Residual Dirichlet Policy Optimization [88.7827307535107]
  Diffusion models (DMs) have achieved state-of-the-art generative performance but suffer from high sampling latency due to their sequential denoising nature.<n>Existing solver-based acceleration methods often face significant image quality degradation under a low-dimensional budget.<n>We propose the Ensemble Parallel Direction solver (dubbed as EPD-EPr), a novel ODE solver that mitigates these errors by incorporating multiple gradient parallel evaluations in each step.
  arXiv  Detail & Related papers  (2025-12-28T05:48:55Z)
• FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models [49.397861654088636]
  We propose a two-step procedure to approximate SVD/QR-based gradient projections into lower-dimensional spaces.<n>We show that our strategy achieves faster runtime and reduced memory usage by up to $25%$ across different model sizes.
  arXiv  Detail & Related papers  (2025-05-23T14:37:00Z)
• Dataset Distillation as Pushforward Optimal Quantization [2.5892916589735457]
  We propose a synthetic training set that achieves similar performance to training on real data, with orders of magnitude less computational requirements.<n>In particular, we link existing disentangled dataset distillation methods to the classical optimal quantization and Wasserstein barycenter problems.<n>We achieve better performance and inter-model generalization on the ImageNet-1K dataset with trivial additional computation, and SOTA performance in higher image-per-class settings.
  arXiv  Detail & Related papers  (2025-01-13T20:41:52Z)
• Stabilizing Linear Passive-Aggressive Online Learning with Weighted Reservoir Sampling [46.01254613933967]
  Online learning methods are still highly effective for high-dimensional streaming data, out-of-core processing, and other throughput-sensitive applications. Many such algorithms rely on fast adaptation to individual errors as a key to their convergence. While such algorithms enjoy low theoretical regret, in real-world deployment they can be sensitive to individual outliers that cause the algorithm to over-correct.
  arXiv  Detail & Related papers  (2024-10-31T03:35:48Z)
• Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation [151.70234052015948]
  We propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7%.
  arXiv  Detail & Related papers  (2022-11-20T15:49:11Z)
• Stable Anderson Acceleration for Deep Learning [0.0]
  Anderson acceleration (AA) is designed to speed-up fixed-point iterations like those arising from the iterative training of DL models. We combine AA with an adaptive moving average procedure that smoothes the oscillations and results in a more regular sequence of descent updates.
  arXiv  Detail & Related papers  (2021-10-26T14:52:16Z)
• Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via GDPA Linearization [59.87663954467815]
  Algorithm unfolding creates an interpretable and parsimonious neural network architecture by implementing each iteration of a model-based algorithm as a neural layer. In this paper, leveraging a recent linear algebraic theorem called Gershgorin disc perfect alignment (GDPA), we unroll a projection-free algorithm for semi-definite programming relaxation (SDR) of a binary graph. Experimental results show that our unrolled network outperformed pure model-based graph classifiers, and achieved comparable performance to pure data-driven networks but using far fewer parameters.
  arXiv  Detail & Related papers  (2021-09-10T07:01:15Z)
• Learning to Optimize Non-Rigid Tracking [54.94145312763044]
  We employ learnable optimizations to improve robustness and speed up solver convergence. First, we upgrade the tracking objective by integrating an alignment data term on deep features which are learned end-to-end through CNN. Second, we bridge the gap between the preconditioning technique and learning method by introducing a ConditionNet which is trained to generate a preconditioner.
  arXiv  Detail & Related papers  (2020-03-27T04:40:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.