ReAttention: Training-Free Infinite Context with Finite Attention Scope

• URL: http://arxiv.org/abs/2407.15176v2
• Date: Sat, 5 Oct 2024 02:09:26 GMT
• Title: ReAttention: Training-Free Infinite Context with Finite Attention Scope
• Authors: Xiaoran Liu, Ruixiao Li, Qipeng Guo, Zhigeng Liu, Yuerong Song, Kai Lv, Hang Yan, Linlin Li, Qun Liu, Xipeng Qiu,
• Abstract summary: Long-context capability of Large Language Models (LLM) has made significant breakthroughs, but the maximum supported context length remains a critical bottleneck limiting their practical applications. We propose bftextReAttention, a training-free approach enabling LLM based on the self-attention mechanism to support an infinite context with a finite attention scope under sufficient memory resources. We validate the performance of ReAttention on the LongBench, L-Eval, and InfiniteBench and demonstrate that it is on par with traditional methods.
• Score: 65.91272939057592
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The long-context capability of the Large Language Models (LLM) has made significant breakthroughs, but the maximum supported context length remains a critical bottleneck limiting their practical applications. The constraint of context length in LLMs arises from the self-attention mechanism, which cannot effectively and efficiently capture the semantic relationships within infinitely long contexts via the limited pre-trained positional information and attention scope. In this work, we propose \textbf{ReAttention}, a training-free approach enabling LLM based on the self-attention mechanism to support an infinite context with a finite attention scope under sufficient memory resources. ReAttention performs the position-agnostic top-$k$ attention before the ordinary position-aware self-attention, freeing LLMs from the length extrapolation issue. We validate the performance of ReAttention on the LongBench, L-Eval, and InfiniteBench and demonstrate that it is on par with traditional methods. Furthermore, we also apply ReAttention on mainstream LLMs, including LLaMA3.1-8B and Mistral-v0.3-7B, enabling them to support context lengths of at least 1M and even expanding the context length of LLaMA3.2-3B-chat by 128$\times$ to 4M without any further training in Needle-In-A-Haystack tests. We also improve the efficiency of ReAttention with Triton and achieve an efficient extrapolation without additional overhead.

Related papers

• Why Does the Effective Context Length of LLMs Fall Short? [68.34573617977013]
  In this work, we introduce ShifTed Rotray position embeddING (STRING) STRING shifts well-trained positions to overwrite the original ineffective positions during inference, enhancing performance within their existing training lengths. Experimental results show that STRING dramatically improves the performance of the latest large-scale models.
  arXiv  Detail & Related papers  (2024-10-24T13:51:50Z)
• Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs [12.250524667536606]
  Large language models (LLMs) still suffer from the challenging extrapolation problem. We conduct a theoretical analysis to better understand why No Position. (NoPE) fails outside its effective range, as well as examining the power of Position. (PE) in this context. We introduce a novel weave PE method, MesaExtrapolation, which utilizes a chunk-based triangular attention matrix and applies Stair-Extrapolation to manage the final chunk.
  arXiv  Detail & Related papers  (2024-10-21T10:39:05Z)
• InfiniPot: Infinite Context Processing on Memory-Constrained LLMs [17.111422610001227]
  InfiniPot is a novel KV cache control framework designed to enable pre-trained Large Language Models to manage extensive sequences efficiently. InfiniPot effectively maintains critical data even without access to future context. This work represents a substantial advancement toward making Large Language Models applicable to a broader range of real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-02T13:09:41Z)
• An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs [54.91212829143966]
  This study explores LLaMA3's capabilities when quantized to low bit-width. We evaluate 10 existing post-training quantization and LoRA-finetuning methods of LLaMA3 on 1-8 bits and diverse datasets. Our experimental results indicate that LLaMA3 still suffers non-negligent degradation in linguistic and visual contexts.
  arXiv  Detail & Related papers  (2024-04-22T10:03:03Z)
• Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding [78.36702055076456]
  This paper introduces Multi-scale Positional. (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of. LLMs to handle relevant information located in the middle of the context.
  arXiv  Detail & Related papers  (2024-03-05T04:58:37Z)
• InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory [93.20588235940453]
  In this paper, we introduce a training-free memory-based method, InfLLM. InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention. Even when the sequence length is scaled to $1,024$K, InfLLM still effectively captures long-distance dependencies.
  arXiv  Detail & Related papers  (2024-02-07T06:50:42Z)
• CLEX: Continuous Length Extrapolation for Large Language Models [68.43814043853347]
  We propose Continuous Length EXtrapolation (CLEX) for Large Language Models (LLMs) CLEX extends the context window to over 4x or almost 8x training length, with no deterioration in performance. Our model trained on a 4k length exhibits competitive performance against state-of-the-art open-source models trained on context lengths up to 32k.
  arXiv  Detail & Related papers  (2023-10-25T08:13:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.