BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models

• URL: http://arxiv.org/abs/2310.01329v2
• Date: Fri, 3 May 2024 05:41:55 GMT
• Title: BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models
• Authors: Qingqing Cao, Sewon Min, Yizhong Wang, Hannaneh Hajishirzi,
• Abstract summary: Retrieval augmentation addresses many critical problems in large language models. Running retrieval-augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages.
• Score: 77.0501668780182
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval augmentation addresses many critical problems in large language models such as hallucination, staleness, and privacy leaks. However, running retrieval-augmented language models (LMs) is slow and difficult to scale due to processing large amounts of retrieved text. We introduce binary token representations (BTR), which use 1-bit vectors to precompute every token in passages, significantly reducing computation during inference. Despite the potential loss of accuracy, our new calibration techniques and training objectives restore performance. Combined with offline and runtime compression, this only requires 127GB of disk space for encoding 3 billion tokens in Wikipedia. Our experiments show that on five knowledge-intensive NLP tasks, BTR accelerates state-of-the-art inference by up to 4x and reduces storage by over 100x while maintaining over 95% task performance.

Related papers

• MrT5: Dynamic Token Merging for Efficient Byte-level Language Models [50.46453950887946]
  This work introduces MrT5 (MergeT5), a more efficient variant of ByT5. MrT5 integrates a token deletion mechanism in its encoder to dynamically shorten the input sequence length. When trained on English text, MrT5 demonstrates the capability to transfer its deletion feature zero-shot across several languages.
  arXiv  Detail & Related papers  (2024-10-28T06:14:12Z)
• Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling [53.58854856174773]
  Speculative decoding is an approach to accelerate inference through a guess-and-verify paradigm. Token Recycling stores candidate tokens in an adjacency matrix and employs a breadth-first search algorithm. It significantly outperforms existing train-free methods by 30% and even a training method by 25%.
  arXiv  Detail & Related papers  (2024-08-16T12:20:56Z)
• Patch-Level Training for Large Language Models [69.67438563485887]
  This paper introduces patch-level training for Large Language Models (LLMs) During patch-level training, we feed the language model shorter sequences of patches and train it to predict the next patch. Following this, the model continues token-level training on the remaining training data to align with the inference mode.
  arXiv  Detail & Related papers  (2024-07-17T15:48:39Z)
• VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections [35.133698935322634]
  Large language models (LLMs) have recently emerged as powerful tools for tackling many language-processing tasks. We identify and characterise the important components needed for effective model convergence using gradient descent. This result leads us to a cheap and memory-efficient algorithm for both fine-tuning and pre-training LLMs.
  arXiv  Detail & Related papers  (2024-05-28T09:23:14Z)
• BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation [13.262366437264188]
  BitDistiller is a framework that synergizes Quantization-Aware Training (QAT) with Knowledge Distillation (KD) to boost the performance of Large Language Models (LLMs) Specifically, BitDistiller first incorporates a tailored asymmetric quantization and clipping technique to maximally preserve the fidelity of quantized weights, and then proposes a novel Confidence-Aware Kullback-Leibler Divergence (CAKLD) objective. Empirical evaluations demonstrate that BitDistiller significantly surpasses existing methods in both 3-bit and 2-bit configurations on general language understanding and complex reasoning benchmarks.
  arXiv  Detail & Related papers  (2024-02-16T12:27:15Z)
• Towards Faster k-Nearest-Neighbor Machine Translation [56.66038663128903]
  k-nearest-neighbor machine translation approaches suffer from heavy retrieve overhead on the entire datastore when decoding each token. We propose a simple yet effective multi-layer perceptron (MLP) network to predict whether a token should be translated jointly by the neural machine translation model and probabilities produced by the kNN.
  arXiv  Detail & Related papers  (2023-12-12T16:41:29Z)
• Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognition [20.926163659469587]
  We propose a new memory augmented lookup dictionary based Transformer architecture for LM. The newly introduced lookup dictionary incorporates rich contextual information in training set, which is vital to correctly predict long-tail tokens. Our proposed method is proved to outperform the baseline Transformer LM by a great margin on both word/character error rate and tail tokens error rate.
  arXiv  Detail & Related papers  (2022-12-30T22:26:57Z)
• Improving language models by retrieving from trillions of tokens [50.42630445476544]
  We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile.
  arXiv  Detail & Related papers  (2021-12-08T17:32:34Z)
• Prune Once for All: Sparse Pre-Trained Language Models [0.6063525456640462]
  We present a new method for training sparse pre-trained Transformer language models by integrating weight pruning and model distillation. These sparse pre-trained models can be used to transfer learning for a wide range of tasks while maintaining their sparsity pattern. We show how the compressed sparse pre-trained models we trained transfer their knowledge to five different downstream natural language tasks with minimal accuracy loss.
  arXiv  Detail & Related papers  (2021-11-10T15:52:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.