LLoCO: Learning Long Contexts Offline

• URL: http://arxiv.org/abs/2404.07979v2
• Date: Thu, 17 Oct 2024 08:54:37 GMT
• Title: LLoCO: Learning Long Contexts Offline
• Authors: Sijun Tan, Xiuyu Li, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E. Gonzalez, Raluca Ada Popa,
• Abstract summary: We propose LLoCO, a novel approach to processing long contexts. LLoCO learns contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA. Our approach extends the effective context window of a 4k token LLaMA2-7B model to handle up to 128k tokens.
• Score: 63.3458260335454
• License:
• Abstract: Processing long contexts remains a challenge for large language models (LLMs) due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation. We propose LLoCO, a novel approach to address this problem by learning contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA. Our method enables an LLM to create a concise representation of the original context and efficiently retrieve relevant information to answer questions accurately. Our approach extends the effective context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We evaluate our approach on several long-context question-answering datasets, demonstrating that LLoCO significantly outperforms in-context learning while using $30\times$ fewer tokens during inference. LLoCO achieves up to $7.62\times$ speed-up during inference and $11.52\times$ higher throughput during finetuning, substantially reduces the cost of long document question answering. This makes it a promising solution for efficient long context processing. Our code is publicly available on https://github.com/jeffreysijuntan/lloco.

Related papers

• LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models [72.71150585370147]
  LongRecipe is an efficient training strategy for extending the context window of large language models. It simulates long-sequence inputs while maintaining training efficiency and significantly improves the model's understanding of long-range dependencies. LongRecipe can utilize long sequences while requiring only 30% of the target context window size, and reduces computational training resource over 85% compared to full sequence training.
  arXiv  Detail & Related papers  (2024-08-31T17:19:30Z)
• QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression [37.08536175557748]
  In this paper, we introduce a novel Query-gUIded aTtention cOmpression (QUITO) method to filter useless information. Specifically, we take a trigger token to calculate the attention distribution of the context in response to the question. We evaluate the QUITO using two widely-used datasets, namely, NaturalQuestions and ASQA.
  arXiv  Detail & Related papers  (2024-08-01T04:28:38Z)
• $\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens [64.08660301017302]
  There is currently a lack of a standardized benchmark to evaluate this long-context capability. $infty$Bench is the first benchmark featuring an average data length surpassing 100K tokens. The results indicate that existing long context LLMs still require significant advancements to effectively process 100K+ context.
  arXiv  Detail & Related papers  (2024-02-21T11:30:29Z)
• Compressing Context to Enhance Inference Efficiency of Large Language Models [26.75216730927996]
  This paper proposes a method called Selective Context to enhance the inference efficiency of large language models (LLMs) We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conversations. Experimental results show that Selective Context significantly reduces memory cost and decreases generation latency.
  arXiv  Detail & Related papers  (2023-10-09T23:03:24Z)
• Retrieval meets Long Context Large Language Models [59.431200671427064]
  Extending context window of large language models (LLMs) is getting popular recently. Retrieval-augmentation versus long context window, which one is better for downstream tasks? Can both methods be combined to get the best of both worlds? Our best model, retrieval-augmented Llama2-70B with 32K context window, outperforms GPT-3.5-turbo-16k and Davinci003 in terms of average score on nine long context tasks.
  arXiv  Detail & Related papers  (2023-10-04T17:59:41Z)
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models [67.58275666573496]
  LongLoRA is an efficient fine-tuning approach that extends the context sizes of pre-trained large language models. We demonstrate strong empirical results on various tasks on Llama2 models from 7B/13B to 70B.
  arXiv  Detail & Related papers  (2023-09-21T17:59:11Z)
• In-context Autoencoder for Context Compression in a Large Language Model [70.7621953091318]
  We propose the In-context Autoencoder (ICAE) to compress a long context into short compact memory slots. ICAE is first pretrained using both autoencoding and language modeling objectives on massive text data.
  arXiv  Detail & Related papers  (2023-07-13T17:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.