Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters

• URL: http://arxiv.org/abs/2406.16758v1
• Date: Mon, 24 Jun 2024 16:06:50 GMT
• Title: Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters
• Authors: Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du-Seong Chang, Se-Young Yun,
• Abstract summary: Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. This paper explores a training recipe of an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup of inference time compared to the previous methods.
• Score: 21.19251212483406
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup of inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation.

Related papers

• Accelerating Multilingual Language Model for Excessively Tokenized Languages [3.5570874721859016]
  tokenizers in large language models (LLMs) often fragment a text into character or Unicode-level tokens in non-Roman alphabetic languages. We introduce a simple yet effective framework to accelerate text generation in such languages.
  arXiv  Detail & Related papers  (2024-01-19T12:26:57Z)
• The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics [74.99898531299148]
  This research examines vocabulary trimming (VT) inspired by restricting embedding entries to the language of interest to bolster time and memory efficiency. We apply two languages to trim the full vocabulary - Unicode-based script filtering and corpus-based selection - to different language families and sizes. It is found that VT reduces the memory usage of small models by nearly 50% and has an upper bound of 25% improvement in generation speed.
  arXiv  Detail & Related papers  (2023-11-16T09:35:50Z)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting [68.19544657508509]
  Large language models (LLMs) are adopted as a fundamental component of language technologies. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt format in few-shot settings. We propose an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights.
  arXiv  Detail & Related papers  (2023-10-17T15:03:30Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)
• Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Models [18.874880342410876]
  We present Jamp, a Japanese benchmark focused on temporal inference. Our dataset includes a range of temporal inference patterns, which enables us to conduct fine-grained analysis. We evaluate the generalization capacities of monolingual/multilingual LMs by splitting our dataset based on tense fragments.
  arXiv  Detail & Related papers  (2023-06-19T07:00:14Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• On the Universality of Deep COntextual Language Models [15.218264849664715]
  Deep Contextual Language Models (LMs) like ELMO, BERT, and their successors dominate the landscape of Natural Language Processing. Multilingual versions of such models like XLM-R and mBERT have given promising results in zero-shot cross-lingual transfer. Due to this initial success, pre-trained models are being used as Universal Language Models'
  arXiv  Detail & Related papers  (2021-09-15T08:00:33Z)
• Efficient Nearest Neighbor Language Models [114.40866461741795]
  Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore. We show how to achieve up to a 6x speed-up in inference speed while retaining comparable performance.
  arXiv  Detail & Related papers  (2021-09-09T12:32:28Z)
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners [23.150999852147283]
  This study proposes a novel pluggable, and efficient approach named DifferentiAble pRompT (DART) It can convert small language models into better few-shot learners without any prompt engineering. A comprehensive evaluation of standard NLP tasks demonstrates that the proposed approach achieves a better few-shot performance.
  arXiv  Detail & Related papers  (2021-08-30T12:29:25Z)
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
  Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages. We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
  arXiv  Detail & Related papers  (2020-12-31T11:37:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.