Speculative Decoding and Beyond: An In-Depth Survey of Techniques

• URL: http://arxiv.org/abs/2502.19732v3
• Date: Tue, 04 Mar 2025 03:46:23 GMT
• Title: Speculative Decoding and Beyond: An In-Depth Survey of Techniques
• Authors: Yunhai Hu, Zining Liu, Zhenyuan Dong, Tianfan Peng, Bradley McDanel, Sai Qian Zhang,
• Abstract summary: Sequential dependencies present a fundamental bottleneck in deploying large-scale autoregressive models.<n>Recent advances in generation-refinement frameworks demonstrate that this trade-off can be significantly mitigated.
• Score: 4.165029665035158
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sequential dependencies present a fundamental bottleneck in deploying large-scale autoregressive models, particularly for real-time applications. While traditional optimization approaches like pruning and quantization often compromise model quality, recent advances in generation-refinement frameworks demonstrate that this trade-off can be significantly mitigated. This survey presents a comprehensive taxonomy of generation-refinement frameworks, analyzing methods across autoregressive sequence tasks. We categorize methods based on their generation strategies (from simple n-gram prediction to sophisticated draft models) and refinement mechanisms (including single-pass verification and iterative approaches). Through systematic analysis of both algorithmic innovations and system-level implementations, we examine deployment strategies across computing environments and explore applications spanning text, images, and speech generation. This systematic examination of both theoretical frameworks and practical implementations provides a foundation for future research in efficient autoregressive decoding.

Related papers

• A Comprehensive Review on Hashtag Recommendation: From Traditional to Deep Learning and Beyond [0.37865171120254354]
  Hashtags, as a fundamental categorization mechanism, play a pivotal role in enhancing content visibility and user engagement. The development of accurate and robust hashtag recommendation systems remains a complex and evolving research challenge. This review article conducts a systematic analysis of hashtag recommendation systems, examining recent advancements across several dimensions.
  arXiv  Detail & Related papers  (2025-03-24T13:40:36Z)
• Beyond Fine-Tuning: A Systematic Study of Sampling Techniques in Personalized Image Generation [2.9631016562930546]
  Balancing the fidelity of the learned concept with its ability for generation in various contexts presents a significant challenge.<n>Existing methods often address this through diverse fine-tuning parameterizations and improved sampling strategies.<n>We propose a decision framework evaluating text alignment, computational constraints, and fidelity objectives to guide strategy selection.
  arXiv  Detail & Related papers  (2025-02-09T13:22:32Z)
• A Survey on Inference Optimization Techniques for Mixture of Experts Models [50.40325411764262]
  Large-scale Mixture of Experts (MoE) models offer enhanced model capacity and computational efficiency through conditional computation.<n> deploying and running inference on these models presents significant challenges in computational resources, latency, and energy efficiency.<n>This survey analyzes optimization techniques for MoE models across the entire system stack.
  arXiv  Detail & Related papers  (2024-12-18T14:11:15Z)
• From Noise to Nuance: Advances in Deep Generative Image Models [8.802499769896192]
  Deep learning-based image generation has undergone a paradigm shift since 2021. Recent developments in Stable Diffusion, DALL-E, and consistency models have redefined the capabilities and performance boundaries of image synthesis. We investigate how enhanced multi-modal understanding and zero-shot generation capabilities are reshaping practical applications across industries.
  arXiv  Detail & Related papers  (2024-12-12T02:09:04Z)
• Boosting CNN-based Handwriting Recognition Systems with Learnable Relaxation Labeling [48.78361527873024]
  We propose a novel approach to handwriting recognition that integrates the strengths of two distinct methodologies. We introduce a sparsification technique that accelerates the convergence of the algorithm and enhances the overall system's performance.
  arXiv  Detail & Related papers  (2024-09-09T15:12:28Z)
• Unifying Self-Supervised Clustering and Energy-Based Models [9.3176264568834]
  We establish a principled connection between self-supervised learning and generative models.<n>We show that our solution can be integrated into a neuro-symbolic framework to tackle a simple yet non-trivial instantiation of the symbol grounding problem.
  arXiv  Detail & Related papers  (2023-12-30T04:46:16Z)
• REX: Rapid Exploration and eXploitation for AI Agents [103.68453326880456]
  We propose an enhanced approach for Rapid Exploration and eXploitation for AI Agents called REX. REX introduces an additional layer of rewards and integrates concepts similar to Upper Confidence Bound (UCB) scores, leading to more robust and efficient AI agent performance.
  arXiv  Detail & Related papers  (2023-07-18T04:26:33Z)
• Learning to Rank in Generative Retrieval [62.91492903161522]
  Generative retrieval aims to generate identifier strings of relevant passages as the retrieval target. We propose a learning-to-rank framework for generative retrieval, dubbed LTRGR. This framework only requires an additional learning-to-rank training phase to enhance current generative retrieval systems.
  arXiv  Detail & Related papers  (2023-06-27T05:48:14Z)
• Generalization Properties of Retrieval-based Models [50.35325326050263]
  Retrieval-based machine learning methods have enjoyed success on a wide range of problems. Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored. We present a formal treatment of retrieval-based models to characterize their generalization ability.
  arXiv  Detail & Related papers  (2022-10-06T00:33:01Z)
• A Multi-criteria Approach to Evolve Sparse Neural Architectures for Stock Market Forecasting [0.0]
  This study proposes a new framework to evolve efficacious yet parsimonious neural architectures for the movement prediction of stock market indices. A new search paradigm, Two-Dimensional Swarms (2DS) is proposed for the multi-criteria neural architecture search. The results of this study convincingly demonstrate that the proposed approach can evolve parsimonious networks with better generalization capabilities.
  arXiv  Detail & Related papers  (2021-11-15T19:44:10Z)
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models [40.08137765886609]
  We show that our model, called a graph structured surrogate model (GSSM), outperforms state-of-the-art methods in predicting environment dynamics. Our approach is able to obtain high returns, while allowing fast execution during deployment by avoiding test time policy gradient optimization.
  arXiv  Detail & Related papers  (2021-02-16T17:21:55Z)
• Reinforcement Learning as Iterative and Amortised Inference [62.997667081978825]
  We use the control as inference framework to outline a novel classification scheme based on amortised and iterative inference. We show that taking this perspective allows us to identify parts of the algorithmic design space which have been relatively unexplored.
  arXiv  Detail & Related papers  (2020-06-13T16:10:03Z)
• Target-Embedding Autoencoders for Supervised Representation Learning [111.07204912245841]
  This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional. We motivate and formalize the general framework of target-embedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets.
  arXiv  Detail & Related papers  (2020-01-23T02:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.