Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation

• URL: http://arxiv.org/abs/2504.18857v1
• Date: Sat, 26 Apr 2025 08:46:10 GMT
• Title: Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation
• Authors: Yi Lu, Wanxu Zhao, Xin Zhou, Chenxin An, Chenglong Wang, Shuo Li, Yuming Yang, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang,
• Abstract summary: Large Language Models (LLMs) often struggle to process and generate coherent context when the number of input tokens exceeds the pre-trained length.<n>Recent advancements in long-context extension have significantly expanded the context window of LLMs.<n>We propose Dimension-Wise Positional Embeddings Manipulation (DPE) to extrapolate the context window of LLMs.
• Score: 60.22622442950905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) often struggle to process and generate coherent context when the number of input tokens exceeds the pre-trained length. Recent advancements in long-context extension have significantly expanded the context window of LLMs but require expensive overhead to train the large-scale models with longer context. In this work, we propose Dimension-Wise Positional Embeddings Manipulation (DPE), a training-free framework to extrapolate the context window of LLMs by diving into RoPE's different hidden dimensions. Instead of manipulating all dimensions equally, DPE detects the effective length for every dimension and finds the key dimensions for context extension. We reuse the original position indices with their embeddings from the pre-trained model and manipulate the key dimensions' position indices to their most effective lengths. In this way, DPE adjusts the pre-trained models with minimal modifications while ensuring that each dimension reaches its optimal state for extrapolation. DPE significantly surpasses well-known baselines such as YaRN and Self-Extend. DPE enables Llama3-8k 8B to support context windows of 128k tokens without continual training and integrates seamlessly with Flash Attention 2. In addition to its impressive extrapolation capability, DPE also dramatically improves the models' performance within training length, such as Llama3.1 70B, by over 18 points on popular long-context benchmarks RULER. When compared with commercial models, Llama 3.1 70B with DPE even achieves better performance than GPT-4-128K.

Related papers

• LongRoPE2: Near-Lossless LLM Context Window Scaling [46.936900701411965]
  LongRoPE2 is a novel approach that extends the effective context window of pre-trained large language models (LLMs) to the target length. This is achieved by three contributions: (1) a hypothesis that insufficient training in higher RoPE dimensions contributes to the persistent out-of-distribution issues observed in existing methods; (2) an effective RoPE rescaling algorithm that adopts evolutionary search guided by "needle-driven" perplexity to address the insufficient training problem; and (3) a mixed context window training approach that fine-tunes model weights to adopt rescaled RoPE for long-context sequences.
  arXiv  Detail & Related papers  (2025-02-27T13:41:07Z)
• Why Does the Effective Context Length of LLMs Fall Short? [68.34573617977013]
  In this work, we introduce ShifTed Rotray position embeddING (STRING) STRING shifts well-trained positions to overwrite the original ineffective positions during inference, enhancing performance within their existing training lengths. Experimental results show that STRING dramatically improves the performance of the latest large-scale models.
  arXiv  Detail & Related papers  (2024-10-24T13:51:50Z)
• LongEmbed: Extending Embedding Models for Long Context Retrieval [87.60404151086715]
  This paper explores context window extension of embedding models, pushing the limit to 32k without requiring additional training. First, we examine the performance of current embedding models for long context retrieval on our newly constructed LongEmbed benchmark. Experiments show that training-free context window extension strategies like positionRo can effectively extend the context window of existing embedding models by several folds.
  arXiv  Detail & Related papers  (2024-04-18T11:29:23Z)
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens [7.833740464264734]
  Current extended context windows are limited to around 128k tokens. LongRoPE extends the context window of pre-trained LLMs to an impressive 2048k tokens.
  arXiv  Detail & Related papers  (2024-02-21T12:30:33Z)
• E^2-LLM: Efficient and Extreme Length Extension of Large Language Models [74.1254067728251]
  We propose an Efficient and Extreme length extension method for Large Language Models, called E 2 -LLM, with only one training procedure and dramatically reduced cost. Comprehensive experimental results on multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on challenging long-context tasks.
  arXiv  Detail & Related papers  (2024-01-13T02:11:20Z)
• CLEX: Continuous Length Extrapolation for Large Language Models [68.43814043853347]
  We propose Continuous Length EXtrapolation (CLEX) for Large Language Models (LLMs) CLEX extends the context window to over 4x or almost 8x training length, with no deterioration in performance. Our model trained on a 4k length exhibits competitive performance against state-of-the-art open-source models trained on context lengths up to 32k.
  arXiv  Detail & Related papers  (2023-10-25T08:13:02Z)
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training [91.99700930388998]
  We propose Positional Skip-wisE training that simulates long inputs using a fixed context window. PoSE greatly reduces memory and time overhead compared with Full-length fine-tuning. We have successfully extended the LLaMA model to 128k tokens using a 2k training context window.
  arXiv  Detail & Related papers  (2023-09-19T08:03:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.