Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding

• URL: http://arxiv.org/abs/2307.05908v2
• Date: Mon, 29 Jul 2024 04:03:22 GMT
• Title: Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding
• Authors: Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, Kangwook Lee,
• Abstract summary: "Predictive Pipelined Decoding (PPD)" is an approach that speeds up greedy decoding in Large Language Models (LLMs) Unlike conventional strategies, PPD employs additional compute resources to parallelize the initiation of subsequent token decoding. We have developed a theoretical framework that allows us to analyze the trade-off between computation and latency.
• Score: 12.49711203027534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents "Predictive Pipelined Decoding (PPD)," an approach that speeds up greedy decoding in Large Language Models (LLMs) while maintaining the exact same output as the original decoding. Unlike conventional strategies, PPD employs additional compute resources to parallelize the initiation of subsequent token decoding during the current token decoding. This method reduces decoding latency and reshapes the understanding of trade-offs in LLM decoding strategies. We have developed a theoretical framework that allows us to analyze the trade-off between computation and latency. Using this framework, we can analytically estimate the potential reduction in latency associated with our proposed method, achieved through the assessment of the match rate, represented as p_correct. The results demonstrate that the use of extra computational resources has the potential to accelerate LLM decoding. Additionally, we implement PPD and conduct preliminary experiments to empirically validate its efficacy, addressing potential practical overheads not covered by theoretical analysis.

Related papers

• A Theoretical Perspective for Speculative Decoding Algorithm [60.79447486066416]
  One effective way to accelerate inference is emphSpeculative Decoding, which employs a small model to sample a sequence of draft tokens and a large model to validate. This paper tackles this gap by conceptualizing the decoding problem via markov chain abstraction and studying the key properties, emphoutput quality and inference acceleration, from a theoretical perspective.
  arXiv  Detail & Related papers  (2024-10-30T01:53:04Z)
• Limitations of the decoding-to-LPN reduction via code smoothing [59.90381090395222]
  The Learning Parity with Noise (LPN) problem underlies several classic cryptographic primitives. This paper attempts to find a reduction from the decoding problem of linear codes, for which several hardness results exist. We characterize the efficiency of the reduction in terms of the parameters of the decoding and problems.
  arXiv  Detail & Related papers  (2024-08-07T12:54:43Z)
• Error correction of parity-encoding-based annealing through post-readout decoding [0.0]
  We show through Monte Carlo simulation that this redundant encoding may be exploited to solve the problems of the inefficiency and computational cost of the parity-encoded scheme. Our findings open up the possibility of parity-encoded schemes for realizing the QA with near-term quantum technologies.
  arXiv  Detail & Related papers  (2024-02-13T22:55:58Z)
• A Thorough Examination of Decoding Methods in the Era of LLMs [72.65956436513241]
  Decoding methods play an indispensable role in converting language models from next-token predictors into practical task solvers. This paper provides a comprehensive and multifaceted analysis of various decoding methods within the context of large language models. Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization.
  arXiv  Detail & Related papers  (2024-02-10T11:14:53Z)
• Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding [46.485363806259265]
  Speculative Decoding has emerged as a novel decoding paradigm for Large Language Models (LLMs) inference. In each decoding step, this method first drafts several future tokens efficiently and then verifies them in parallel. This paper presents a comprehensive overview and analysis of this promising decoding paradigm.
  arXiv  Detail & Related papers  (2024-01-15T17:26:50Z)
• Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster [61.83949316226113]
  FastCoT is a model-agnostic framework based on parallel decoding. We show that FastCoT saves inference time by nearly 20% with only a negligible performance drop compared to the regular approach.
  arXiv  Detail & Related papers  (2023-11-14T15:56:18Z)
• Graph Neural Networks for Enhanced Decoding of Quantum LDPC Codes [6.175503577352742]
  We propose a differentiable iterative decoder for quantum low-density parity-check (LDPC) codes. The proposed algorithm is composed of classical belief propagation (BP) decoding stages and intermediate graph neural network (GNN) layers.
  arXiv  Detail & Related papers  (2023-10-26T19:56:25Z)
• Coded Distributed Computing with Partial Recovery [56.08535873173518]
  We introduce a novel coded matrix-vector multiplication scheme, called coded computation with partial recovery (CCPR) CCPR reduces both the computation time and the decoding complexity by allowing a trade-off between the accuracy and the speed of computation. We then extend this approach to distributed implementation of more general computation tasks by proposing a coded communication scheme with partial recovery.
  arXiv  Detail & Related papers  (2020-07-04T21:34:49Z)
• A PDD Decoder for Binary Linear Codes With Neural Check Polytope Projection [43.97522161614078]
  We propose a PDD algorithm to address the fundamental polytope based maximum likelihood (ML) decoding problem. We also propose to integrate machine learning techniques into the most time-consuming part of the PDD decoding algorithm. We present a specially designed neural CPP (N CPP) algorithm to decrease the decoding latency.
  arXiv  Detail & Related papers  (2020-06-11T07:57:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.