ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation

• URL: http://arxiv.org/abs/2606.29869v1
• Date: Mon, 29 Jun 2026 07:05:56 GMT
• Title: ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation
• Abstract summary: Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs)<n>We analyze the complementary roles of forward and reverse KL divergence (FKL/RKL) in distribution alignment from theoretical and empirical perspectives.<n>We propose a reinforcement-learning-based adaptive KL-weighted distillation framework, in which a policy network dynamically assigns weights to FKL and RKL based on teacher-student distributional characteristics.
• Score: 5.7913606195390885
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization. To address this, we analyze the complementary roles of forward and reverse KL divergence (FKL/RKL) in distribution alignment from theoretical and empirical perspectives. We then propose a reinforcement-learning-based adaptive KL-weighted distillation framework, in which a policy network dynamically assigns weights to FKL and RKL based on teacher-student distributional characteristics, guided by immediate reward signals to achieve dual alignment on principal and long-tail modes. Extensive experiments demonstrate consistent improvements across Rouge-L and BertScore metrics, surpassing greedy heuristics by 0.4-0.6 points and outperforming other baseline methods on diverse benchmarks.

Related papers

• Rethinking Reverse KL as Adaptive Entropy Distillation [18.957879192719762]
  We propose textbfAdaptive Entropy Distillation (AED), which uses the teacher's entropy to dynamically calibrate token-level imitation strength.<n>AED achieves superior overall performance and generally improves teacher--student distributional and entropy alignment.
  arXiv  Detail & Related papers  (2026-08-05T04:27:37Z)
• Density-Ratio Losses for Post-Hoc Learning to Defer [61.81110533670321]
  We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions.<n>We define deferral via the density-ratio between a model's and an expert's ideals.<n>Deferral decisions are then made by thresholding the scorer, allowing deferral rates to be adjusted without retraining.
  arXiv  Detail & Related papers  (2026-05-19T08:56:23Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation [6.256286464835529]
  We propose Diversity-aware RKL (DRKL), which removes this gradient effect and strengthens non-target supervision while preserving the optimization benefits of RKL.<n>DRKL consistently outperforms FKL, RKL, and other state-of-the-art distillation objectives, achieving better performance and a superior fidelity-diversity trade-off.
  arXiv  Detail & Related papers  (2026-03-31T20:39:47Z)
• Reinforcement-aware Knowledge Distillation for LLM Reasoning [63.53679456364683]
  Reinforcement learning (RL) post-training has recently driven gains in long chain-of-thought reasoning large language models (LLMs)<n>Most existing knowledge distillation methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization.<n>We propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update.
  arXiv  Detail & Related papers  (2026-02-26T00:20:39Z)
• A Comedy of Estimators: On KL Regularization in RL Training of LLMs [81.7906270099878]
  reinforcement learning (RL) can substantially improve the reasoning performance of large language models (LLMs)<n>The RL objective for LLM training involves a regularization term, which is the reverse Kullback-Leibler (KL) divergence between the trained policy and the reference policy.<n>Recent works show that prevailing practices for incorporating KL regularization do not provide correct gradients for stated objectives, creating a discrepancy between the objective and its implementation.<n>We study the gradients of several estimators configurations, revealing how design choices shape gradient bias.
  arXiv  Detail & Related papers  (2025-12-26T04:20:58Z)
• Biased Teacher, Balanced Student [0.0]
  Long-Tailed Knowledge Distillation (LTKD) is a novel framework tailored for class-imbalanced scenarios.<n>Experiments on CIFAR-100-LT, TinyImageNet-LT, and ImageNet-LT show that LTKD consistently outperforms existing KD methods.
  arXiv  Detail & Related papers  (2025-06-23T10:46:44Z)
• KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning [72.53466291156604]
  We present textbfKDRL, a textitunified post-training framework that jointly optimize a reasoning model through teacher supervision (KD) and self-exploration (RL)<n>We first formulate a unified objective that integrates GRPO and KD, and systematically explore how different KL approximations, KL coefficients, and reward-guided KD strategies affect the overall post-training dynamics and performance.
  arXiv  Detail & Related papers  (2025-06-02T19:46:41Z)
• ToDi: Token-wise Distillation via Fine-Grained Divergence Control [9.958797874295355]
  Token-wise Distillation (ToDi) is a novel method that adaptively combines Forward KL and Reverse KL per token using a sigmoid-based weighting function.<n>ToDi consistently outperforms recent distillation baselines using uniform or less granular strategies.
  arXiv  Detail & Related papers  (2025-05-22T06:51:16Z)
• Direct Preference Knowledge Distillation for Large Language Models [73.50849692633953]
  We propose Direct Preference Knowledge Distillation (DPKD) for large language models (LLMs)<n>We re-formulate KD of LLMs into two stages: first optimizing and objective consisting of implicit reward and reverse KL divergence.<n>We prove the value and effectiveness of the introduced implicit reward and output preference in KD through experiments and theoretical analysis.
  arXiv  Detail & Related papers  (2024-06-28T09:23:40Z)
• Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models [18.870276152694245]
  Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to compress Large Language Models (LLMs)<n>Contrary to prior assertions, reverse Kullback-Leibler (RKL) divergence is mode-seeking and thus preferable over the mean-seeking forward Kullback-Leibler (FKL) divergence.<n>We propose a simple yet effective Adaptive Kullback-Leiber (AKL) divergence method, which adaptively allocates weights to combine FKL and RKL.
  arXiv  Detail & Related papers  (2024-04-03T11:40:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.