XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference

• URL: http://arxiv.org/abs/2405.17755v1
• Date: Tue, 28 May 2024 02:12:35 GMT
• Title: XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference
• Authors: Shengnan Wang, Youhui Bai, Lin Zhang, Pingyi Zhou, Shixiong Zhao, Gong Zhang, Sen Wang, Renhai Chen, Hua Xu, Hongwei Sun,
• Abstract summary: We propose an efficient training free framework, named XL3M, which enables the LLMs trained on short sequences to reason extremely long sequence without any further training or fine-tuning. Evaluations on comprehensive benchmarks show the superiority of XL3M.
• Score: 25.669630896777484
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Length generalization failure problem, namely the large language model (LLM) fails to generalize to texts longer than its maximum training length, greatly restricts the application of LLM in the scenarios with streaming long inputs. To address this problem, the existing methods either require substantial costs or introduce precision loss. In this paper, we empirically find that the accuracy of the LLM's prediction is highly correlated to its certainty. Based on this, we propose an efficient training free framework, named XL3M (it means extra-long large language model), which enables the LLMs trained on short sequences to reason extremely long sequence without any further training or fine-tuning. Under the XL3M framework, the input context will be firstly decomposed into multiple short sub-contexts, where each sub-context contains an independent segment and a common ``question'' which is a few tokens from the end of the original context. Then XL3M gives a method to measure the relevance between each segment and the ``question'', and constructs a concise key context by splicing all the relevant segments in chronological order. The key context is further used instead of the original context to complete the inference task. Evaluations on comprehensive benchmarks show the superiority of XL3M. Using our framework, a Llama2-7B model is able to reason 20M long sequences on an 8-card Huawei Ascend 910B NPU machine with 64GB memory per card.

Related papers

• LLM$\times$MapReduce: Simplified Long-Sequence Processing using Large Language Models [73.13933847198395]
  We propose a training-free framework for processing long texts, utilizing a divide-and-conquer strategy to achieve comprehensive document understanding. The proposed LLM$times$MapReduce framework splits the entire document into several chunks for LLMs to read and then aggregates the intermediate answers to produce the final output.
  arXiv  Detail & Related papers  (2024-10-12T03:13:44Z)
• LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models [72.71150585370147]
  LongRecipe is an efficient training strategy for extending the context window of large language models. It simulates long-sequence inputs while maintaining training efficiency and significantly improves the model's understanding of long-range dependencies. LongRecipe can utilize long sequences while requiring only 30% of the target context window size, and reduces computational training resource over 85% compared to full sequence training.
  arXiv  Detail & Related papers  (2024-08-31T17:19:30Z)
• FocusLLM: Scaling LLM's Context by Parallel Decoding [16.642675785000176]
  FocusLLM is a framework designed to extend the context length of any decoder-only LLM. FocusLLM processes long text inputs by dividing them into chunks based on the model's original context length. It appends the local context to each chunk as a prompt to extract essential information from each chunk based on a novel parallel decoding mechanism.
  arXiv  Detail & Related papers  (2024-08-21T16:11:59Z)
• InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory [93.20588235940453]
  In this paper, we introduce a training-free memory-based method, InfLLM. InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention. Even when the sequence length is scaled to $1,024$K, InfLLM still effectively captures long-distance dependencies.
  arXiv  Detail & Related papers  (2024-02-07T06:50:42Z)
• E^2-LLM: Efficient and Extreme Length Extension of Large Language Models [74.1254067728251]
  We propose an Efficient and Extreme length extension method for Large Language Models, called E 2 -LLM, with only one training procedure and dramatically reduced cost. Comprehensive experimental results on multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on challenging long-context tasks.
  arXiv  Detail & Related papers  (2024-01-13T02:11:20Z)
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training [91.99700930388998]
  We propose Positional Skip-wisE training that simulates long inputs using a fixed context window. PoSE greatly reduces memory and time overhead compared with Full-length fine-tuning. We have successfully extended the LLaMA model to 128k tokens using a 2k training context window.
  arXiv  Detail & Related papers  (2023-09-19T08:03:38Z)
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding [58.20031627237889]
  LongBench is the first bilingual, multi-task benchmark for long context understanding. It comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese)
  arXiv  Detail & Related papers  (2023-08-28T11:53:40Z)
• Giraffe: Adventures in Expanding Context Lengths in LLMs [7.8327063299618]
  We show that linear scaling is the best method for extending context length. We also discover promising extrapolation capabilities in the truncated basis. To support further research in this area, we release three new 13B parameter long-context models.
  arXiv  Detail & Related papers  (2023-08-21T17:30:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.