Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning

• URL: http://arxiv.org/abs/2606.25488v1
• Date: Wed, 24 Jun 2026 07:17:05 GMT
• Title: Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning
• Abstract summary: Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments.<n>We propose IF-Beta, an efficient data pruning framework that combines influence functions with a learnable sampling policy.<n>Experiments on CIFAR-10/100 and ImageNet show that IF-Beta consistently outperforms other baselines across a wide range of pruning ratios.
• Score: 39.145926634579105
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments. Yet the computational overhead of the distillation process itself is often overlooked, raising the question of whether a better student model can be obtained with less data and less compute via data pruning. However, existing data pruning methods are not designed for KD: some introduce substantial overhead, such as obtaining training dynamics through retraining, while others rely on heuristic selection rules that fail to capture what KD actually requires, often resulting in suboptimal subsets. To address these issues, we propose IF-Beta, an efficient data pruning framework that combines influence functions with a learnable sampling policy. Empirically, we first demonstrate that influence functions can serve as an effective and efficient estimator of sample impact in KD settings, where only a pretrained teacher is available. Building on this, our sampling policy is specifically parameterized by a Beta distribution, whose highly flexible two-parameter family allows the policy to adapt to diverse pruning regimes rather than being tied to fixed heuristic forms. Next, we formulate KD pruning as optimizing this policy through a bilevel objective, where the inner loop operates in the teacher feature space with a KD-aligned objective, enabling fast proxy training, while the outer loop updates the policy parameters to maximize distillation performance. This design ensures that IF-Beta is both computationally efficient and inherently aligned with the goals of KD. Extensive experiments on CIFAR-10/100 and ImageNet show that IF-Beta consistently outperforms other baselines across a wide range of pruning ratios. Remarkably, IF-Beta enables students trained on less data and less compute to surpass the performance of students distilled on the full dataset.

Related papers

• OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework [30.771327347534996]
  OrderDP aims to obtain stable, unbiased, and near-lossless training acceleration with theoretical guarantees.<n>We evaluate OrderDP against comprehensive baselines on CIFAR-10, CIFAR-100, and ImageNet-1K.
  arXiv  Detail & Related papers  (2026-06-07T11:11:51Z)
• Data-Efficient RLVR via Off-Policy Influence Guidance [84.60336960383867]
  This work proposes a theoretically-grounded approach using influence functions to estimate the contribution of each data point to the learning objective.<n>We develop textbfCurriculum textbfRL with textbfOff-textbfPolicy textInfluence guidance (textbfCROPI), a multi-stage RL framework that iteratively selects the most influential data for the current policy.
  arXiv  Detail & Related papers  (2025-10-30T13:40:52Z)
• Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models [0.0]
  Knowledge Distillation (KD) is a technique for compressing large language models (LLMs) into compact, efficient student models.<n>We propose Selective Reflection Distillation (SRD), a novel data curation framework.<n>As a plug-and-play enhancement, SRD consistently improves distillation outcomes across diverse white-box KD approaches.
  arXiv  Detail & Related papers  (2025-08-08T08:55:53Z)
• Online Knowledge Distillation with Reward Guidance [3.9160947065896803]
  This work studies knowledge distillation (KD) for large language models (LLMs) through preference optimization.<n>We propose a reward-guided imitation learning framework for sequential KD, formulating a min-max optimization problem between the policy and reward model (RM) to minimize the performance gap between the student and teacher policies.
  arXiv  Detail & Related papers  (2025-05-25T02:56:18Z)
• Active Data Curation Effectively Distills Large-Scale Multimodal Models [66.23057263509027]
  Knowledge distillation (KD) is the de facto standard for compressing large-scale models into smaller ones.<n>In this work we explore an alternative, yet simple approach -- active data curation as effective distillation for contrastive multimodal pretraining.<n>Our simple online batch selection method, ACID, outperforms strong KD baselines across various model-, data- and compute-configurations.
  arXiv  Detail & Related papers  (2024-11-27T18:50:15Z)
• Condensed Data Expansion Using Model Inversion for Knowledge Distillation [39.800536851433776]
  We propose a method that expands condensed datasets using model inversion.<n>By creating synthetic data that complements the condensed samples, we enrich the training set and better approximate the underlying data distribution.<n>Our method demonstrates significant gains in KD accuracy compared to using condensed datasets alone.
  arXiv  Detail & Related papers  (2024-08-25T14:43:27Z)
• Direct Preference Knowledge Distillation for Large Language Models [73.50849692633953]
  We propose Direct Preference Knowledge Distillation (DPKD) for large language models (LLMs)<n>We re-formulate KD of LLMs into two stages: first optimizing and objective consisting of implicit reward and reverse KL divergence.<n>We prove the value and effectiveness of the introduced implicit reward and output preference in KD through experiments and theoretical analysis.
  arXiv  Detail & Related papers  (2024-06-28T09:23:40Z)
• Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning [3.1423836318272773]
  Knowledge distillation (KD) improves the performance of efficient and lightweight models.<n>Most existing KD techniques rely on Kullback-Leibler (KL) divergence.<n>We propose a Robustness-Reinforced Knowledge Distillation (R2KD) that leverages correlation distance and network pruning.
  arXiv  Detail & Related papers  (2023-11-23T11:34:48Z)
• Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation [51.66271681532262]
  Online Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming to improve the performance of any deep neural model in an online manner. We utilize k-nn non-parametric density estimation technique for estimating the unknown probability distributions of the data samples in the output feature space.
  arXiv  Detail & Related papers  (2021-08-26T14:01:04Z)
• MixKD: Towards Efficient Distillation of Large-scale Language Models [129.73786264834894]
  We propose MixKD, a data-agnostic distillation framework, to endow the resulting model with stronger generalization ability. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the error and the empirical error. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.
  arXiv  Detail & Related papers  (2020-11-01T18:47:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.