PACED: Distillation and Self-Distillation at the Frontier of Student Competence

• URL: http://arxiv.org/abs/2603.11178v2
• Date: Mon, 16 Mar 2026 12:54:11 GMT
• Title: PACED: Distillation and Self-Distillation at the Frontier of Student Competence
• Abstract summary: Paced is a framework that concentrates distillation on the zone of proximal development.<n>On instruction-tuned models with reverse KL, gains are exceeding baselines.<n>A forward-KL-then-reverse-KL schedule yields the strongest results in our setting.
• Score: 17.384089089363382
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard LLM distillation wastes compute on two fronts: problems the student has already mastered (near-zero gradients) and problems far beyond its reach (incoherent gradients that erode existing capabilities). We show that this waste is not merely intuitive but structurally inevitable: the gradient signal-to-noise ratio in distillation provably vanishes at both pass-rate extremes. This theoretical observation leads to Paced, a framework that concentrates distillation on the zone of proximal development -- the frontier of a student model's competence -- via a principled pass-rate weight $w(p) = p^α(1 - p)^β$ derived from the boundary-vanishing structure of distillation gradients. Key results: (1) Theory: We prove that the Beta kernel $w(p) = p^α(1-p)^β$ is a leading-order weight family arising from the SNR structure of distillation, and that it is minimax-robust -- under bounded multiplicative misspecification, worst-case efficiency loss is only $O(δ^2)$. (2)Distillation: On distillation from a larger teacher to a smaller student model with forward KL, Paced achieves significant gain over the base model, while keeping benchmark forgetting at a low level. (3)Self-distillation: On instruction-tuned models with reverse KL, gains are exceeding baselines as well. (4)Two-stage synergy: A forward-KL-then-reverse-KL schedule yields the strongest results in our setting, reaching substantial improvements on standard reasoning benchmarks -- supporting a mode-coverage-then-consolidation interpretation of the distillation process. All configurations require only student rollouts to estimate pass rates, need no architectural changes, and are compatible with any KL direction.

Related papers

• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation [59.43605629311855]
  We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits.<n>Our primary contribution is therefore a gradient-level characterization of reverse-KL OPD trained with the K2 estimator, with SuRe as one empirical instantiation.
  arXiv  Detail & Related papers  (2026-08-26T11:14:42Z)
• The Distributional View of Knowledge Distillation [13.841028090842869]
  Token-level knowledge distillation matches two conditional distributions per position.<n>We develop a distributional view in which the teacher is represented not by a single softened output but by a family of multi-temperature views.
  arXiv  Detail & Related papers  (2026-08-15T13:03:31Z)
• On-Policy Delta Distillation [61.76889440384448]
  On-policy distillation is an alternative post-training method in reinforcement learning.<n>We introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution.<n>We show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$2$)
  arXiv  Detail & Related papers  (2026-07-16T16:07:19Z)
• Diffusion Fine-tuning with Rewarded Moment Matching Distillation [38.86104469934903]
  Rewarded Moment Matching Distillation (RMMD) is a novel framework that simultaneously distills diffusion models and maximizes a reward function.<n>We apply RMMD to GenCast, a state-of-the-art weather forecasting model, to distill it while optimizing the Continuous Ranked Probability Score (CRPS) metric.<n>The resulting distilled model achieves a 7.5x speedup while outperforming the teacher model on 93% of target weather variables, and being better calibrated.
  arXiv  Detail & Related papers  (2026-06-29T15:00:56Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models [9.025382179940767]
  coarse-to-fine distillation framework with LInear FiTtingbased distillation (LIFT) and Piecewise Local Adaptive Coefficient Estimation (PLACE)<n>Experiments show LIFT and PLACE is effective across diffusion spaces (image/latent), backbones (U-Net/DiT), tasks (unconditional/conditional), datasets, and even extends to flow-based models such as MMDiT (SD3)
  arXiv  Detail & Related papers  (2026-05-19T12:03:53Z)
• On the Generalization of Knowledge Distillation: An Information-Theoretic View [5.248154928825152]
  We model teacher and student training as coupled processes and introduce a distillation divergence.<n>We develop a loss-sharpness-aware bound with an explicit tightness regime, showing that the teacher's local flatness can strictly tighten the bound.<n>In a linear Gaussian case study, the distillation divergence admits an interpretable decomposition into bias, variance, and rank-bottleneck costs.
  arXiv  Detail & Related papers  (2026-05-13T08:10:05Z)
• Distillation Traps and Guards: A Calibration Knob for LLM Distillability [54.90137955363471]
  We propose a post-hoc calibration method that enables control over a teacher's distillability via reinforcement fine-tuning.<n>Our objective combines task utility, KL anchor, and across-tokenizer calibration reward.<n> Experiments show that students distilled from distillable teachers outperform SFT and KD baselines.
  arXiv  Detail & Related papers  (2026-04-21T01:22:35Z)
• HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation [19.13058927652914]
  We propose Hindsight Entropy-Assisted Learning (HEAL), an RL-free framework designed to bridge this reasoning gap.<n>HEAL synergizes three core modules: (1) Guided Entropy-Assisted Repair (GEAR), an active intervention mechanism that detects critical reasoning breakpoints via entropy dynamics and injects targeted hindsight hints to repair broken trajectories; (2) Perplexity-Uncertainty Ratio Estimator (PURE), a rigorous filtering protocol that decouples genuine cognitive breakthroughs from spurious shortcuts; and (3) Progressive Answer-guided Curriculum Evolution (PACE), a three-stage distillation strategy that organizes training from foundational alignment to
  arXiv  Detail & Related papers  (2026-03-11T03:12:10Z)
• Why Alignment Must Precede Distillation: A Minimal Working Explanation [50.784080714897776]
  We show that the standard KD -> Align workflow diminishes the model's capacity to align rare yet desirable behaviors.<n>We demonstrate that alignment must first be performed on a high-recall reference before distillation.
  arXiv  Detail & Related papers  (2025-09-28T06:12:19Z)
• Knowledge distillation through geometry-aware representational alignment [3.901188865224763]
  We show that existing feature distillation methods cannot capture the feature structure, even under zero loss.<n>We then motivate the use of Procrustes distance and the Frobenius norm of Feature Gram Matrix, distances already common in the context of measuring representational alignment.<n>We show that feature distillation through our method showcases statistically significant improvement in distillation performance across language models families.
  arXiv  Detail & Related papers  (2025-09-27T09:59:46Z)
• Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis [65.77083310980896]
  We propose Adrial Distribution Matching (ADM) to align latent predictions between real and fake score estimators for score distillation.<n>Our proposed method achieves superior one-step performance on SDXL compared to DMD2 while consuming less GPU time.<n>Additional experiments that apply multi-step ADM distillation on SD3-Medium, SD3.5-Large, and CogVideoX set a new benchmark towards efficient image and video synthesis.
  arXiv  Detail & Related papers  (2025-07-24T16:45:05Z)
• Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis [20.2271205957037]
  Hyper-SD is a novel framework that amalgamates the advantages of ODE Trajectory Preservation and Reformulation. We introduce Trajectory Segmented Consistency Distillation to progressively perform consistent distillation within pre-defined time-step segments. We incorporate human feedback learning to boost the performance of the model in a low-step regime and mitigate the performance loss incurred by the distillation process.
  arXiv  Detail & Related papers  (2024-04-21T15:16:05Z)
• Knowledge Distillation Performs Partial Variance Reduction [93.6365393721122]
  Knowledge distillation is a popular approach for enhancing the performance of ''student'' models. The underlying mechanics behind knowledge distillation (KD) are still not fully understood. We show that KD can be interpreted as a novel type of variance reduction mechanism.
  arXiv  Detail & Related papers  (2023-05-27T21:25:55Z)
• Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation [72.70058049274664]
  We present Referee, a novel framework for sentence summarization that can be trained reference-free (i.e., requiring no gold summaries for supervision) Our work is the first to demonstrate that reference-free, controlled sentence summarization is feasible via the conceptual framework of Symbolic Knowledge Distillation.
  arXiv  Detail & Related papers  (2022-10-25T07:07:54Z)
• Aligning Logits Generatively for Principled Black-Box Knowledge Distillation [49.43567344782207]
  Black-Box Knowledge Distillation (B2KD) is a formulated problem for cloud-to-edge model compression with invisible data and models hosted on the server. We formalize a two-step workflow consisting of deprivatization and distillation. We propose a new method Mapping-Emulation KD (MEKD) that distills a black-box cumbersome model into a lightweight one.
  arXiv  Detail & Related papers  (2022-05-21T02:38:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.