Training Domain Draft Models for Speculative Decoding: Best Practices and Insights

• URL: http://arxiv.org/abs/2503.07807v2
• Date: Tue, 25 Mar 2025 22:17:33 GMT
• Title: Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
• Authors: Fenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker, Avinash Ravichandran, Swayambhoo Jain, Changran Hu,
• Abstract summary: When adapting speculative decoding to domain-specific target models, the acceptance rate of the generic draft model drops significantly due to domain shift.<n>We compare white-box and black-box distillation approaches and explore their effectiveness in various data accessibility scenarios.<n>We find that synthetic data can effectively align draft models and achieve 80% to 93% of the performance of training on historical user queries.
• Score: 16.68232264939302
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speculative decoding is an effective method for accelerating inference of large language models (LLMs) by employing a small draft model to predict the output of a target model. However, when adapting speculative decoding to domain-specific target models, the acceptance rate of the generic draft model drops significantly due to domain shift. In this work, we systematically investigate knowledge distillation techniques for training domain draft models to improve their speculation accuracy. We compare white-box and black-box distillation approaches and explore their effectiveness in various data accessibility scenarios, including historical user queries, curated domain data, and synthetically generated alignment data. Our experiments across Function Calling, Biology, and Chinese domains show that offline distillation consistently outperforms online distillation by 11% to 25%, white-box distillation surpasses black-box distillation by 2% to 10%, and data scaling trends hold across domains. Additionally, we find that synthetic data can effectively align draft models and achieve 80% to 93% of the performance of training on historical user queries. These findings provide practical guidelines for training domain-specific draft models to improve speculative decoding efficiency.

Related papers

• DAViD: Data-efficient and Accurate Vision Models from Synthetic Data [6.829390872619486]
  We demonstrate that it is possible to train models on much smaller but high-fidelity synthetic datasets.<n>Our models require only a fraction of the cost of training and inference when compared with foundational models of similar accuracy.
  arXiv  Detail & Related papers  (2025-07-21T08:17:41Z)
• Efficient Data Selection at Scale via Influence Distillation [53.03573620682107]
  This paper introduces Influence Distillation, a mathematicallyjustified framework for data selection.<n>By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data.<n>Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to $3.5times$ faster selection.
  arXiv  Detail & Related papers  (2025-05-25T09:08:00Z)
• OpenCodeReasoning: Advancing Data Distillation for Competitive Coding [61.15402517835137]
  We build a supervised fine-tuning (SFT) dataset to achieve state-of-the-art coding capability results in models of various sizes. Our models use only SFT to achieve 61.8% on LiveCodeBench and 24.6% on CodeContests, surpassing alternatives trained with reinforcement learning.
  arXiv  Detail & Related papers  (2025-04-02T17:50:31Z)
• Dataset Distillation via Committee Voting [21.018818924580877]
  We introduce $bf C$ommittee $bf V$oting for $bf D$ataset $bf D$istillation (CV-DD)<n>CV-DD is a novel approach that leverages the collective wisdom of multiple models or experts to create high-quality distilled datasets.
  arXiv  Detail & Related papers  (2025-01-13T18:59:48Z)
• Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales [10.397502254316645]
  We propose a two-phase scheme to ensure double-correct predictions. First, we curate a new dataset that offers structured rationales for visual recognition tasks. Second, we propose a rationale-informed optimization method to guide the model in disentangling and localizing visual evidence.
  arXiv  Detail & Related papers  (2024-10-31T18:33:39Z)
• Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient [52.2669490431145]
  PropEn is inspired by'matching', which enables implicit guidance without training a discriminator. We show that training with a matched dataset approximates the gradient of the property of interest while remaining within the data distribution.
  arXiv  Detail & Related papers  (2024-05-28T11:30:19Z)
• Exploring the potential of prototype-based soft-labels data distillation for imbalanced data classification [0.0]
  Main goal is to push further the performance of prototype-based soft-labels distillation in terms of classification accuracy. Experimental studies trace the capability of the method to distill the data, but also the opportunity to act as an augmentation method.
  arXiv  Detail & Related papers  (2024-03-25T19:15:19Z)
• Towards Adversarially Robust Dataset Distillation by Curvature Regularization [11.02948004359488]
  dataset distillation (DD) allows datasets to be distilled to fractions of their original size while preserving the rich distributional information. Recent research in this area has been focusing on improving the accuracy of models trained on distilled datasets. We propose a new method that achieves this goal by incorporating curvature regularization into the distillation process with much less computational overhead than standard adversarial training.
  arXiv  Detail & Related papers  (2024-03-15T06:31:03Z)
• Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation [96.92250565207017]
  We study the data efficiency and selection for the dataset distillation task. By re-formulating the dynamics of distillation, we provide insight into the inherent redundancy in the real dataset. We find the most contributing samples based on their causal effects on the distillation.
  arXiv  Detail & Related papers  (2023-05-28T06:53:41Z)
• Post-training Model Quantization Using GANs for Synthetic Data Generation [57.40733249681334]
  We investigate the use of synthetic data as a substitute for the calibration with real data for the quantization method. We compare the performance of models quantized using data generated by StyleGAN2-ADA and our pre-trained DiStyleGAN, with quantization using real data and an alternative data generation method based on fractal images.
  arXiv  Detail & Related papers  (2023-05-10T11:10:09Z)
• Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation [51.66271681532262]
  Online Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming to improve the performance of any deep neural model in an online manner. We utilize k-nn non-parametric density estimation technique for estimating the unknown probability distributions of the data samples in the output feature space.
  arXiv  Detail & Related papers  (2021-08-26T14:01:04Z)
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones [40.33419553042038]
  We propose to improve existing baseline networks via knowledge distillation from off-the-shelf pre-trained big powerful models. Our solution performs distillation by only driving prediction of the student model consistent with that of the teacher model. We empirically find that such simple distillation settings perform extremely effective, for example, the top-1 accuracy on ImageNet-1k validation set of MobileNetV3-large and ResNet50-D can be significantly improved.
  arXiv  Detail & Related papers  (2021-03-10T09:32:44Z)
• Distilling Interpretable Models into Human-Readable Code [71.11328360614479]
  Human-readability is an important and desirable standard for machine-learned model interpretability. We propose to train interpretable models using conventional methods, and then distill them into concise, human-readable code. We describe a piecewise-linear curve-fitting algorithm that produces high-quality results efficiently and reliably across a broad range of use cases.
  arXiv  Detail & Related papers  (2021-01-21T01:46:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.