The Distributional View of Knowledge Distillation

• URL: http://arxiv.org/abs/2608.15215v1
• Date: Sat, 15 Aug 2026 13:03:31 GMT
• Title: The Distributional View of Knowledge Distillation
• Abstract summary: Token-level knowledge distillation matches two conditional distributions per position.<n>We develop a distributional view in which the teacher is represented not by a single softened output but by a family of multi-temperature views.
• Score: 13.841028090842869
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Token-level knowledge distillation (KD) matches two conditional distributions per position, yet the standard objectives compare them pointwise: a Kullback-Leibler gradient is blind to which wrong token receives probability mass. We develop a distributional view in which the teacher is represented not by a single softened output but by a family of multi-temperature views - marginals of the annealing path of its logits - and the student is trained against a geometry-aware aggregate of these views under an embedding-based ground cost. We formalize the resulting design space (mixtures, log-linear pooling, entropic Wasserstein barycenters, and a debiased Sinkhorn-divergence flagship in hub and path forms), prove an exact collapse result showing log-linear pooling of tempered views is equivalent to a single temperature, and give a multi-marginal Schrodinger-bridge reading that yields falsifiable predictions. On instruction-tuned Pythia pairs, experiments yield three empirical laws: (i) dispersion law - the benefit of multi-temperature aggregation grows monotonically with the effective temperature dispersion of the views, not with their number; (ii) dispersed views unlock the aggregation operator - the barycenter separates from the arithmetic mixture exactly when transport-based aggregation starts to beat averaging; and (iii) two-regime picture governed by the ceiling gap $Γ=\mathrm{PPL}_{\mathrm{SFT}}-\mathrm{PPL}_{T}$: when the fine-tuned teacher barely beats a supervised student the gentle transport objective is the best KD loss but no KD beats supervised fine-tuning, whereas at a real ceiling the ranking inverts - and the sign of the fidelity-generalization correlation flips. We argue that "which distillation loss is the best" is not a fixed property of the loss but a function of $Γ$.

Related papers

• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation [59.43605629311855]
  We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits.<n>Our primary contribution is therefore a gradient-level characterization of reverse-KL OPD trained with the K2 estimator, with SuRe as one empirical instantiation.
  arXiv  Detail & Related papers  (2026-08-26T11:14:42Z)
• The Loss Floor of Denoising Score Matching: Fisher Geometry from Schrödinger Bridges [4.661816556403101]
  We show that for a general corruption kernel under mild regularity assumptions, it is exactly the trace of the Fisher--Rao metric of the conditional endpoint family.<n>We derive the result from a Schr"odinger bridge variational principle, in which the ideal objective arises as excess path-space relative entropy.
  arXiv  Detail & Related papers  (2026-08-24T23:48:06Z)
• Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence [67.12978375116599]
  We study an alternative objective: the reverse Fisher divergence, where the expectation is taken with respect to the student distribution.<n>We prove global convergence guarantees under a $widetilde(1)$-separation assumption on the target means.<n>Our rely on a new Lyapunov-based analysis of the gradient descent dynamics, showing that the reverse Fisher divergence has a much more favorable optimization landscape than the forward Fisher divergence.
  arXiv  Detail & Related papers  (2026-06-18T07:34:33Z)
• Information-Geometric Decomposition of Generalization Error in Unsupervised Learning [0.0]
  We decompose the Kullback--Leibler generalization error (GE) of unsupervised learning into three non-negative components: model error, data bias, and variance.<n>The decomposition is exact for any e-flat model class.
  arXiv  Detail & Related papers  (2026-04-14T06:23:18Z)
• Generative Drifting is Secretly Score Matching: a Spectral and Variational Perspective [33.2175112734242]
  Generative Modeling via Drifting has recently achieved state-of-the-art one-step image generation through a kernel-based drift operator.<n>We make the following observation: emphunder a Gaussian kernel, the drift operator is exactly a score difference on smoothed distributions.
  arXiv  Detail & Related papers  (2026-03-10T17:30:35Z)
• A Unified View of Drifting and Score-Based Models [92.51878018325031]
  We make our relationship to the score-matching principle behind diffusion models precise by showing that drifting admits a score-based formulation on kernel-smoothed distributions.<n>We derive an exact decomposition for general radial kernels, and for the Laplace kernel we prove rigorous error bounds showing that drifting remains an accurate proxy for score matching in low-temperature and high-dimensional regimes.
  arXiv  Detail & Related papers  (2026-03-08T07:41:36Z)
• Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation [50.19746127327559]
  We propose a new tail-aware divergence that decouples the contribution of the teacher model's top-K predicted probabilities from that of lower-probability predictions.<n> Experimental results demonstrate that our modified distillation method yields competitive performance in both pre-training and supervised distillation of decoder models.
  arXiv  Detail & Related papers  (2026-02-24T11:54:06Z)
• Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidance [51.532841645285835]
  We study machine unlearning in large generative models by framing the task as density ratio estimation to a target distribution.<n>We show it can fail to faithfully unlearn with finite samples when the forget set represents a sharp, concentrated data distribution.<n>We introduce Temper-Then-Tilt Unlearning (T3-Unlearning), which freezes the base model and applies a two-step inference procedure.
  arXiv  Detail & Related papers  (2026-02-10T19:08:40Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Kernel-Smoothed Scores for Denoising Diffusion: A Bias-Variance Study [3.265950484493743]
  Diffusion models can be prone to memorization.<n>Regularization on the score has the same effect as increasing the size of the training dataset.<n>This perspective highlights two regularization mechanisms taking place in denoising diffusions.
  arXiv  Detail & Related papers  (2025-05-28T20:22:18Z)
• A Unified Analysis for Finite Weight Averaging [50.75116992029417]
  Averaging iterations of Gradient Descent (SGD) have achieved empirical success in training deep learning models, such as Weight Averaging (SWA), Exponential Moving Average (EMA), and LAtest Weight Averaging (LAWA) In this paper, we generalize LAWA as Finite Weight Averaging (FWA) and explain their advantages compared to SGD from the perspective of optimization and generalization.
  arXiv  Detail & Related papers  (2024-11-20T10:08:22Z)
• The Copycat Perceptron: Smashing Barriers Through Collective Learning [3.55026004901472]
  We analyze a general setting in which thermal noise is present that affects each student's generalization performance. We find that the coupling of replicas leads to a bend of the phase diagram towards smaller values of $alpha$. These results provide additional analytic and numerical evidence for the recently conjectured Bayes-optimal property of Replicated Simulated Annealing.
  arXiv  Detail & Related papers  (2023-08-07T17:51:09Z)
• Debiased Sinkhorn barycenters [110.79706180350507]
  Entropy regularization in optimal transport (OT) has been the driver of many recent interests for Wasserstein metrics and barycenters in machine learning. We show how this bias is tightly linked to the reference measure that defines the entropy regularizer. We propose debiased Wasserstein barycenters that preserve the best of both worlds: fast Sinkhorn-like iterations without entropy smoothing.
  arXiv  Detail & Related papers  (2020-06-03T23:06:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.