You Only Cache Once: Decoder-Decoder Architectures for Language Models

• URL: http://arxiv.org/abs/2405.05254v2
• Date: Thu, 9 May 2024 14:12:45 GMT
• Title: You Only Cache Once: Decoder-Decoder Architectures for Language Models
• Authors: Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei,
• Abstract summary: We introduce a decoder-decoder architecture, YOCO, for large language models. YOCO only caches key-value pairs once. The overall model behaves like a decoder-only Transformer, although YOCO only caches once.
• Score: 132.4064488592704
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once. It consists of two components, i.e., a cross-decoder stacked upon a self-decoder. The self-decoder efficiently encodes global key-value (KV) caches that are reused by the cross-decoder via cross-attention. The overall model behaves like a decoder-only Transformer, although YOCO only caches once. The design substantially reduces GPU memory demands, yet retains global attention capability. Additionally, the computation flow enables prefilling to early exit without changing the final output, thereby significantly speeding up the prefill stage. Experimental results demonstrate that YOCO achieves favorable performance compared to Transformer in various settings of scaling up model size and number of training tokens. We also extend YOCO to 1M context length with near-perfect needle retrieval accuracy. The profiling results show that YOCO improves inference memory, prefill latency, and throughput by orders of magnitude across context lengths and model sizes. Code is available at https://aka.ms/YOCO.

Related papers

• Deliberation in Latent Space via Differentiable Cache Augmentation [48.228222586655484]
  We show that a frozen large language model can be augmented with an offline coprocessor that operates on the model's key-value (kv) cache. This coprocessor augments the cache with a set of latent embeddings designed to improve the fidelity of subsequent decoding. We show experimentally that when a cache is augmented, the decoder achieves lower perplexity on numerous subsequent tokens.
  arXiv  Detail & Related papers  (2024-12-23T18:02:25Z)
• Are Decoder-Only Large Language Models the Silver Bullet for Code Search? [32.338318300589776]
  This study presents the first systematic exploration of decoder-only large language models for code search. We evaluate nine state-of-the-art decoder-only models using two fine-tuning methods, two datasets, and three model sizes. Our findings reveal that fine-tuned CodeGemma significantly outperforms encoder-only models like UniXcoder.
  arXiv  Detail & Related papers  (2024-10-29T17:05:25Z)
• Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation [67.85309547416155]
  A powerful architecture for universal segmentation relies on transformers that encode multi-scale image features and decode object queries into mask predictions. With efficiency being a high priority for scaling such models, we observed that the state-of-the-art method Mask2Former uses 50% of its compute only on the transformer encoder. This is due to the retention of a full-length token-level representation of all backbone feature scales at each encoder layer.
  arXiv  Detail & Related papers  (2024-04-23T01:34:20Z)
• ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference [70.36083572306839]
  This paper proposes a new training and inference paradigm for re-ranking. We finetune a pretrained encoder-decoder model using in the form of document to query generation. We show that this encoder-decoder architecture can be decomposed into a decoder-only language model during inference.
  arXiv  Detail & Related papers  (2022-04-25T06:26:29Z)
• UniXcoder: Unified Cross-Modal Pre-training for Code Representation [65.6846553962117]
  We present UniXcoder, a unified cross-modal pre-trained model for programming language. We propose a one-to-one mapping method to transform AST in a sequence structure that retains all structural information from the tree. We evaluate UniXcoder on five code-related tasks over nine datasets.
  arXiv  Detail & Related papers  (2022-03-08T04:48:07Z)
• Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding [57.08875260900373]
  We propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC) SAD aggressively decodes as many tokens as possible in parallel instead of always decoding only one token in each step to improve computational parallelism. Experiments in both English and Chinese GEC benchmarks show that aggressive decoding could yield the same predictions but with a significant speedup for online inference.
  arXiv  Detail & Related papers  (2021-06-09T10:30:59Z)
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers [149.78470371525754]
  We treat semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR) SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes.
  arXiv  Detail & Related papers  (2020-12-31T18:55:57Z)
• Efficient Wait-k Models for Simultaneous Machine Translation [46.01342928010307]
  Simultaneous machine translation consists in starting output generation before the entire input sequence is available. Wait-k decoders offer a simple but efficient approach for this problem. We investigate the behavior of wait-k decoding in low resource settings for spoken corpora using IWSLT datasets.
  arXiv  Detail & Related papers  (2020-05-18T11:14:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.