Faster Cascades via Speculative Decoding

• URL: http://arxiv.org/abs/2405.19261v2
• Date: Mon, 21 Oct 2024 18:12:44 GMT
• Title: Faster Cascades via Speculative Decoding
• Authors: Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, Sanjiv Kumar,
• Abstract summary: Cascades and speculative decoding are approaches to improving language models' inference efficiency. We propose new speculative cascading techniques that implement their deferral rule through speculative execution. We show that our approach yields better cost quality trade-offs than cascading and speculative decoding baselines.
• Score: 66.16909847419198
• License:
• Abstract: Cascades and speculative decoding are two common approaches to improving language models' inference efficiency. Both approaches involve interleaving models of different sizes, but via fundamentally distinct mechanisms: cascades employ a deferral rule that invokes the larger model only for "hard" inputs, while speculative decoding uses speculative execution to primarily invoke the larger model in parallel verification mode. These mechanisms offer different benefits: empirically, cascades offer better cost-quality trade-offs, often even outperforming the large model, while theoretically, speculative decoding offers a guarantee of quality-neutrality. In this paper, we leverage the best of both these approaches by designing new speculative cascading techniques that implement their deferral rule through speculative execution. We characterize the optimal deferral rule for our speculative cascades, and employ a plug-in approximation to the optimal rule. Experiments with Gemma and T5 models on a range of language benchmarks show that our approach yields better cost quality trade-offs than cascading and speculative decoding baselines.

Related papers

• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• Stochastic Two Points Method for Deep Model Zeroth-order Optimization [32.459322001738144]
  This paper introduces an efficient Two-Point (S2P) approach within the gradient-free regime. We present the theoretical convergence properties of S2P under the general and relaxed smoothness assumptions. We show that VS2P is highly effective in optimizing objectives for deep models.
  arXiv  Detail & Related papers  (2024-02-02T18:39:40Z)
• Cascade Speculative Drafting for Even Faster LLM Inference [25.642604897018852]
  Speculative decoding improves the efficiency of large language model (LLM) inference. We introduce Cascade Speculative Drafting (CS Drafting), a speculative execution algorithm that incorporates two types of cascades. CS Drafting achieves up to an 81 percent additional speedup over speculative decoding in our experiments.
  arXiv  Detail & Related papers  (2023-12-18T18:59:46Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)
• Towards Better Certified Segmentation via Diffusion Models [62.21617614504225]
  segmentation models can be vulnerable to adversarial perturbations, which hinders their use in critical-decision systems like healthcare or autonomous driving. Recently, randomized smoothing has been proposed to certify segmentation predictions by adding Gaussian noise to the input to obtain theoretical guarantees. In this paper, we address the problem of certifying segmentation prediction using a combination of randomized smoothing and diffusion models.
  arXiv  Detail & Related papers  (2023-06-16T16:30:39Z)
• SlimSeg: Slimmable Semantic Segmentation with Boundary Supervision [54.16430358203348]
  We propose a simple but effective slimmable semantic segmentation (SlimSeg) method, which can be executed at different capacities during inference. We show that our proposed SlimSeg with various mainstream networks can produce flexible models that provide dynamic adjustment of computational cost and better performance.
  arXiv  Detail & Related papers  (2022-07-13T14:41:05Z)
• Twist Decoding: Diverse Generators Guide Each Other [116.20780037268801]
  We introduce Twist decoding, a simple and general inference algorithm that generates text while benefiting from diverse models. Our method does not assume the vocabulary, tokenization or even generation order is shared.
  arXiv  Detail & Related papers  (2022-05-19T01:27:53Z)
• BODAME: Bilevel Optimization for Defense Against Model Extraction [10.877450596327407]
  We consider an adversarial setting to prevent model extraction under the assumption that will make best guess on the service provider's attacker. We formulate a surrogate model using the predictions of the true model. We give a tractable transformation and an algorithm for more complicated models that are learned by using gradient descent-based algorithms.
  arXiv  Detail & Related papers  (2021-03-11T17:08:31Z)
• Learning Deep-Latent Hierarchies by Stacking Wasserstein Autoencoders [22.54887526392739]
  We propose a novel approach to training models with deep-latent hierarchies based on Optimal Transport. We show that our method enables the generative model to fully leverage its deep-latent hierarchy, avoiding the well known "latent variable collapse" issue of VAEs.
  arXiv  Detail & Related papers  (2020-10-07T15:04:20Z)
• Proximal Mapping for Deep Regularization [15.48377586806766]
  Underpinning the success of deep learning is effective regularizations that allow a variety of priors in data to be modeled. We propose inserting proximal mapping as a new layer to the deep network, which directly and explicitly produces well regularized hidden layer outputs. The resulting technique is shown well connected to kernel warping and dropout, and novel algorithms were developed for robust temporal learning and multiview modeling.
  arXiv  Detail & Related papers  (2020-06-14T07:04:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.