ReAD: Reinforcement-Guided Capability Distillation for Large Language Models

• URL: http://arxiv.org/abs/2605.11290v1
• Date: Mon, 11 May 2026 22:17:51 GMT
• Title: ReAD: Reinforcement-Guided Capability Distillation for Large Language Models
• Abstract summary: We study capability distillation under a fixed token budget and identify two consistent patterns.<n>We propose ReAD, a Reinforcement-guided cApability Distillation framework.
• Score: 69.00808591939524
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Capability distillation applies knowledge distillation to selected model capabilities, aiming to compress a large language model (LLM) into a smaller one while preserving the abilities needed for a downstream task. However, most existing methods treat capabilities as independent training targets and overlook how improving one capability can reshape the student's broader capability profile, especially when multiple abilities jointly determine task success. We study capability distillation under a fixed token budget and identify two consistent patterns: distillation induces systematic, budget-dependent cross-capability transfer, and additional budget often brings limited task-relevant gains while sometimes degrading other useful abilities. Building on these insights, we propose ReAD, a Reinforcement-guided cApability Distillation framework that explicitly accounts for capability interdependence. ReAD first infers task-essential capabilities, then generates capability-targeted supervision on the fly, and finally uses an uncertainty-aware contextual bandit to adaptively allocate the distillation budget based on expected utility gains. Extensive experiments show that ReAD improves downstream utility under the same token budget while reducing harmful spillover and wasted distillation effort compared to strong baselines. Our code is publicly available at https://github.com/LabRAI/ReAD.

Related papers

• Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents [11.967142343676171]
  We propose BCSD (Bi Context Self-Distillation), a framework that combines self-distillation and reinforcement learning to train agents to use external skills more effectively.<n> Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales.
  arXiv  Detail & Related papers  (2026-08-10T12:53:06Z)
• Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization [22.479229036646156]
  We propose a Privileged Tutoring Distillation Policy Optimization framework for Reinforcement Learning with Verifiable Rewards (RLVR)<n> PTD-PO provides dense guidance without exposing the answer to the student policy.<n> Experiments on LVLMs ranging from 2B to 8B parameters show that PTD-PO consistently outperforms RLVR and distillation baselines.
  arXiv  Detail & Related papers  (2026-06-05T07:43:22Z)
• When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards [49.25249414962884]
  We propose Reinforcement Learning with Active Verifiable Rewards (RLAVR)<n>RLAVR actively acquires ground-truth labels for a small set of selected samples and integrates them with pseudo-labels.<n>Building on this, we introduce Correction-Aware Reliability Estimation for RLAVR (CARE)
  arXiv  Detail & Related papers  (2026-05-25T13:55:12Z)
• SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs [55.46289074417954]
  reinforcement learning with verifiable rewards (RLVR) reliably improves pass@1 on reasoning tasks, yet often fails to yield comparable gains in pass@k.<n>We argue that a central structural constraint arises from reverse-KL regularization, which stabilizes training but inherently anchors the policy to the reference distribution.<n>We propose SAGE, a principled framework that enables controllable empirical support expansion by reshaping the reverse-KL anchor distribution itself.
  arXiv  Detail & Related papers  (2026-05-15T07:42:21Z)
• Artificial Jagged Intelligence as Uneven Optimization Energy Allocation Capability Concentration, Redistribution, and Optimization Governance [3.719121868494767]
  Artificial Jagged Intelligence (AJI) denotes a recurring pattern in which large learning systems exhibit strong local capabilities while remaining weak or brittle in other domains.<n>This paper develops a formal theory of AJI as uneven allocation of optimization pressure.
  arXiv  Detail & Related papers  (2026-05-02T12:37:28Z)
• Distillation Traps and Guards: A Calibration Knob for LLM Distillability [54.90137955363471]
  We propose a post-hoc calibration method that enables control over a teacher's distillability via reinforcement fine-tuning.<n>Our objective combines task utility, KL anchor, and across-tokenizer calibration reward.<n> Experiments show that students distilled from distillable teachers outperform SFT and KD baselines.
  arXiv  Detail & Related papers  (2026-04-21T01:22:35Z)
• Beyond Distribution Sharpening: The Importance of Task Rewards [20.11905685439596]
  We compare distribution sharpening and task-reward-based learning, utilizing RL as a tool to implement both paradigms.<n>Experiments using Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct and Qwen3-4B-Instruct-2507 on math datasets confirm that sharpening yields limited gains.
  arXiv  Detail & Related papers  (2026-04-17T17:17:55Z)
• Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation [50.19746127327559]
  We propose a new tail-aware divergence that decouples the contribution of the teacher model's top-K predicted probabilities from that of lower-probability predictions.<n> Experimental results demonstrate that our modified distillation method yields competitive performance in both pre-training and supervised distillation of decoder models.
  arXiv  Detail & Related papers  (2026-02-24T11:54:06Z)
• Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield [54.328202401611264]
  Diffusion model distillation has emerged as a powerful technique for creating efficient few-step and single-step generators.<n>We show that the primary driver of few-step distillation is not distribution matching, but a previously overlooked component we identify as CFG Augmentation (CA)<n>We propose principled modifications to the distillation process, such as decoupling the noise schedules for the engine and the regularizer, leading to further performance gains.
  arXiv  Detail & Related papers  (2025-11-27T18:24:28Z)
• BARD: budget-aware reasoning distillation [25.725960386304646]
  Long Chain-of-Thought (CoT) distillation effectively transfers reasoning capability to smaller language models.<n>We propose bftextBudget-Aware Reasoning Distillation (BARD), a novel framework that simultaneously distills reasoning capability and enables fine-grained control over the reasoning length.
  arXiv  Detail & Related papers  (2025-11-03T11:30:18Z)
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness [49.72591739116668]
  Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs)<n>However, prevalent methods like GRPO often fail when task difficulty exceeds the model's capacity, leading to reward sparsity and inefficient training.<n>We propose HINT: Helping Ineffective rollouts Navigate Towards effectiveness, an adaptive hinting framework.
  arXiv  Detail & Related papers  (2025-10-10T13:42:03Z)
• Augmenting Unsupervised Reinforcement Learning with Self-Reference [63.68018737038331]
  Humans possess the ability to draw on past experiences explicitly when learning new tasks. We propose the Self-Reference (SR) approach, an add-on module explicitly designed to leverage historical information. Our approach achieves state-of-the-art results in terms of Interquartile Mean (IQM) performance and Optimality Gap reduction on the Unsupervised Reinforcement Learning Benchmark.
  arXiv  Detail & Related papers  (2023-11-16T09:07:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.