SODA: Semi On-Policy Black-Box Distillation for Large Language Models

• URL: http://arxiv.org/abs/2604.03873v2
• Date: Sun, 12 Apr 2026 23:46:23 GMT
• Title: SODA: Semi On-Policy Black-Box Distillation for Large Language Models
• Abstract summary: Black-box knowledge distillation for large language models presents a strict trade-off.<n>Semi evaluations across four compact Qwen2.5 and Llama-3 models validate this semi on-policy paradigm.<n>It achieves this superior distillation quality while training 10 times faster, consuming 27% less peak GPU memory, and completely eliminating adversarial instability.
• Score: 19.748920924563546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Black-box knowledge distillation for large language models presents a strict trade-off. Simple off-policy methods (e.g., sequence-level knowledge distillation) struggle to correct the student's inherent errors. Fully on-policy methods (e.g., Generative Adversarial Distillation) solve this via adversarial training but introduce well-known training instability and crippling computational overhead. To address this dilemma, we propose SODA (Semi On-policy Distillation with Alignment), a highly efficient alternative motivated by the inherent capability gap between frontier teachers and much smaller base models. Because a compact student model's natural, zero-shot responses are almost strictly inferior to the powerful teacher's targets, we can construct a highly effective contrastive signal simply by pairing the teacher's optimal response with a one-time static snapshot of the student's outputs. This demonstrates that exposing the small student to its own static inferior behaviors is sufficient for high-quality distribution alignment, eliminating the need for costly dynamic rollouts and fragile adversarial balancing. Extensive evaluations across four compact Qwen2.5 and Llama-3 models validate this semi on-policy paradigm. SODA matches or outperforms the state-of-the-art methods on 15 out of 16 benchmark results. More importantly, it achieves this superior distillation quality while training 10 times faster, consuming 27% less peak GPU memory, and completely eliminating adversarial instability.

Related papers

• Weak-to-Strong On-Policy Distillation [21.651421588571296]
  We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a framework that improves the strong student by distilling from multiple weak models.<n>Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker.
  arXiv  Detail & Related papers  (2026-07-28T20:31:48Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation [49.98710755440242]
  Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities.<n>We reveal a critical inefficiency in this approach: up to 69% of the prompts in standard chart / document reasoning datasets are effectively zero-delta.<n>We propose a staged synthesis pipeline that repurposes existing datasets as seeds, actively targeting student failure modes to produce better prompts.
  arXiv  Detail & Related papers  (2026-05-15T02:04:12Z)
• SOD: Step-wise On-policy Distillation for Small Language Model Agents [32.49707795291693]
  Tool-integrated reasoning is difficult to scale to small language models.<n>Recently, on-policy distillation has gained popularity by supplying dense token-level supervision from a teacher.<n>We propose SOD, a step-wise on-policy distillation framework for small language model agents.
  arXiv  Detail & Related papers  (2026-05-08T13:30:42Z)
• Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion [84.20493238687187]
  We introduce a novel framework that moves beyond simple mimicry to instill a deeper conceptual understanding.<n>underlinetextitFirst, to address pattern memorization, Explanatory Inversion (EI) generates targeted explanatory probes''<n>underlinetextitSecond, to improve generalization, Explanatory GRPO (texttEXGRPO) uses a reinforcement learning algorithm with a novel Dialogue Structure Utility Bonus.
  arXiv  Detail & Related papers  (2026-02-26T23:01:46Z)
• OVD: On-policy Verbal Distillation [47.727229201069555]
  On-policy Verbal Distillation (OVD) is a memory-efficient framework that replaces token-level probability matching with trajectory matching.<n>OVD dramatically reduces memory consumption while enabling on-policy distillation from teacher models with verbal feedback.
  arXiv  Detail & Related papers  (2026-01-29T16:48:14Z)
• Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning [48.041170200238206]
  We introduce DASD-4B-Thinking, a lightweight yet highly capable, fully open-source reasoning model.<n>It achieves SOTA performance among open-source models of comparable scale across challenging benchmarks in mathematics, scientific reasoning, and code generation.
  arXiv  Detail & Related papers  (2026-01-14T02:43:17Z)
• AdaSwitch: Adaptive Switching Generation for Knowledge Distillation [58.647880811071495]
  Small language models (SLMs) are crucial for applications with strict latency and computational constraints.<n>We propose AdaSwitch, a novel approach that combines on-policy and off-policy generation at the token level.<n>AdaSwitch consistently improves accuracy, offering a practical and effective method for distilling SLMs with acceptable additional overhead.
  arXiv  Detail & Related papers  (2025-10-09T06:38:37Z)
• Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method [1.5839621757142595]
  We propose a novel adaptive distillation framework that dynamically augments training data in regions of high student model loss.<n>Our method identifies underperforming regions in the embedding space and generates targeted synthetic examples to guide student learning.
  arXiv  Detail & Related papers  (2025-08-20T15:29:00Z)
• Unleashing the Power of One-Step Diffusion based Image Super-Resolution via a Large-Scale Diffusion Discriminator [81.81748032199813]
  Diffusion models have demonstrated excellent performance for real-world image super-resolution (Real-ISR)<n>We propose a new One-Step textbfDiffusion model with a larger-scale textbfDiscriminator for SR.<n>Our discriminator is able to distill noisy features from any time step of diffusion models in the latent space.
  arXiv  Detail & Related papers  (2024-10-05T16:41:36Z)
• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models [62.5501109475725]
  Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. This paper introduces Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
  arXiv  Detail & Related papers  (2024-09-19T07:05:26Z)
• Indirect Gradient Matching for Adversarial Robust Distillation [15.184564265850382]
  Adrial training significantly improves adversarial robustness, but superior performance is primarily attained with large models.<n>Existing adversarial distillation (AD) methods leverage the teacher's logits as a guide.<n>We propose a distillation module termed Indirect Gradient Distillation Module (IGDM) that indirectly matches the student's input gradient with that of the teacher.
  arXiv  Detail & Related papers  (2023-12-06T04:32:38Z)
• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping [64.54271680071373]
  Diffusion models have demonstrated excellent potential for generating diverse images. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few. We present a novel technique called BOOT, that overcomes limitations with an efficient data-free distillation algorithm.
  arXiv  Detail & Related papers  (2023-06-08T20:30:55Z)
• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.