The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works

• URL: http://arxiv.org/abs/2605.26246v1
• Date: Mon, 25 May 2026 18:12:54 GMT
• Title: The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works
• Abstract summary: We show that mixing hard and soft labels consistently yields better results.<n>It comes from reduced exposure bias, the mismatch between training and inference distributions.<n>We develop a family of Bridge-Garden hybrid supervision methods.
• Score: 84.74671323132492
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge distillation (KD) transfers knowledge from a large teacher model to a smaller student. In language modeling, the student is trained either on tokens sampled from the teacher (hard labels) or the teacher's full next-token distribution (soft labels). Despite soft labels appear strictly richer, we find that mixing hard and soft labels consistently yields better results. Crucially, we show that this gain cannot be explained by closer teacher matching during training. Instead, it comes from reduced exposure bias, the mismatch between training and inference distributions. To explain this phenomenon, we introduce the Bridge-Garden Decomposition theory, which categorizes generation steps into two types: Bridges, where the next token must be exact, and Gardens, where it can be flexible. We show that hard-only KD excels in Bridges by avoiding risky deviations, while soft-only KD preserves diversity in Gardens. A hybrid strategy handles both cases and, as a result, reduces exposure bias across the sequence. Guided by this theory, we develop a family of Bridge-Garden hybrid supervision methods that adaptively balance hard and soft labels. Across a primary suite of seven teacher-student pairs (including Qwen, Llama, Gemma, and DeepSeek) and benchmarks in reasoning and coding, our approach outperforms divergence-based and on-policy KD baselines while reducing training cost by 9.7x, enabling efficient model compression. Code is available at https://github.com/ghwang-s/bridge_garden_hybrid_kd_release.

Related papers

• Mismatch Matters: On-Policy Distillation Beyond Token Agreement [27.396104103072844]
  We show that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens.<n>We propose TIDE, which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass.
  arXiv  Detail & Related papers  (2026-08-10T16:53:14Z)
• X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation [19.064775527374618]
  Cross-tokenizer knowledge distillation allows a student model to learn from teachers with vocabularies.<n>Logit-based methods either use only the correct-token probability, missing the full 'dark knowledge' in the teacher's distribution.<n>We propose X-Token, an approach with two complementary loss formulations targeting these issues.
  arXiv  Detail & Related papers  (2026-05-20T19:59:31Z)
• Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training [20.04756350098974]
  In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated to the model and reward density where it is most informative.<n>We identify a reward-density principle that governs this allocation: sparse sequence-level reward is most useful on models that can explore and discover better behavior, while dense token-level teacher supervision is better suited for compressing that behavior into a smaller deployment model.
  arXiv  Detail & Related papers  (2026-05-12T17:57:48Z)
• ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $α$-$β$-Divergence [89.630486749083]
  Knowledge Distillation (KD) transfers knowledge from a large teacher model to a smaller student model.<n>The core challenge in KD lies in balancing two mode-concentration effects.<n>We propose ABKD, a generic framework with $alpha$$beta$-divergence.
  arXiv  Detail & Related papers  (2025-05-07T16:48:49Z)
• A Dual-Space Framework for General Knowledge Distillation of Large Language Models [98.73585104789217]
  Knowledge distillation (KD) is a promising solution to compress large language models (LLMs) by transferring their knowledge to smaller models.<n>The current white-box KD framework exhibits two limitations.<n>We propose a dual-space knowledge distillation (DSKD) framework that unifies the prediction heads of the teacher and the student models for KD.
  arXiv  Detail & Related papers  (2025-04-15T17:38:47Z)
• Unbiased Knowledge Distillation for Recommendation [66.82575287129728]
  Knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge to supervise the learning of a compact student model. We find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation.
  arXiv  Detail & Related papers  (2022-11-27T05:14:03Z)
• Continuous Soft Pseudo-Labeling in ASR [32.19655911858698]
  Continuous pseudo-labeling (PL) algorithms have emerged as a powerful strategy for semi-supervised learning in speech recognition. We find that soft-labels targets can lead to training divergence, with the model collapsing to a degenerate token distribution per frame.
  arXiv  Detail & Related papers  (2022-11-11T05:16:18Z)
• Black-box Few-shot Knowledge Distillation [55.27881513982002]
  Knowledge distillation (KD) is an efficient approach to transfer the knowledge from a large "teacher" network to a smaller "student" network. We propose a black-box few-shot KD method to train the student with few unlabeled training samples and a black-box teacher. We conduct extensive experiments to show that our method significantly outperforms recent SOTA few/zero-shot KD methods on image classification tasks.
  arXiv  Detail & Related papers  (2022-07-25T12:16:53Z)
• ALM-KD: Knowledge Distillation with noisy labels via adaptive loss mixing [25.49637460661711]
  Knowledge distillation is a technique where the outputs of a pretrained model are used for training a student model in a supervised setting. We tackle this problem via the use of an adaptive loss mixing scheme during KD. We demonstrate performance gains obtained using our approach in the standard KD setting as well as in multi-teacher and self-distillation settings.
  arXiv  Detail & Related papers  (2022-02-07T14:53:22Z)
• Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation [9.157410884444312]
  Knowledge distillation (KD) has been investigated to design efficient neural architectures. We show that the KL divergence loss focuses on the logit matching when tau increases and the label matching when tau goes to 0. We show that sequential distillation can improve performance and that KD, particularly when using the KL divergence loss with small tau, mitigates the label noise.
  arXiv  Detail & Related papers  (2021-05-19T04:40:53Z)
• Distilling Double Descent [65.85258126760502]
  Distillation is the technique of training a "student" model based on examples that are labeled by a separate "teacher" model. We show, that, even when the teacher model is highly over parameterized, and provides emphhard labels, using a very large held-out unlabeled dataset can result in a model that outperforms more "traditional" approaches.
  arXiv  Detail & Related papers  (2021-02-13T02:26:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.