Resonance RoPE: Improving Context Length Generalization of Large Language Models

• URL: http://arxiv.org/abs/2403.00071v2
• Date: Mon, 10 Jun 2024 13:30:34 GMT
• Title: Resonance RoPE: Improving Context Length Generalization of Large Language Models
• Authors: Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, Bang Liu,
• Abstract summary: This paper addresses the challenge of train-short-test-long (TSTL) scenarios in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE) We introduce Resonance RoPE, a novel approach designed to narrow the generalization gap in TSTL scenarios. We present PosGen, a new synthetic benchmark specifically designed for fine-grained behavior analysis in TSTL scenarios.
• Score: 37.749813693281254
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper addresses the challenge of train-short-test-long (TSTL) scenarios in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE), where models pre-trained on shorter sequences face difficulty with out-of-distribution (OOD) token positions in longer sequences. We introduce Resonance RoPE, a novel approach designed to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions, significantly improving the model performance without additional online computational costs. Furthermore, we present PosGen, a new synthetic benchmark specifically designed for fine-grained behavior analysis in TSTL scenarios, aiming to isolate the constantly increasing difficulty of token generation on long contexts from the challenges of recognizing new token positions. Our experiments on synthetic tasks show that after applying Resonance RoPE, Transformers recognize OOD position better and more robustly. Our extensive LLM experiments also show superior performance after applying Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on both upstream language modeling tasks and a variety of downstream long-text applications.

Related papers

• HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation [19.42279057349193]
  positional encodings (PEs) are designed to exhibit long-term decay, based on an entrenched and long-standing inductive opinion. We argue that long-term decay is outdated in the era of LLMs, as LLMs are now applied to tasks demanding precise retrieval of in-context information.
  arXiv  Detail & Related papers  (2024-10-28T17:01:52Z)
• Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Prompting [21.04933334040135]
  We introduce the Self-Prompting framework, a novel method designed to fully harness the embedded RE knowledge within Large Language Models. Our framework employs a three-stage diversity approach to prompt LLMs, generating multiple synthetic samples that encapsulate specific relations from scratch. Experimental evaluations on benchmark datasets show our approach outperforms existing LLM-based zero-shot RE methods.
  arXiv  Detail & Related papers  (2024-10-02T01:12:54Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Towards Inducing Document-Level Abilities in Standard Multilingual Neural Machine Translation Models [4.625277907331917]
  This work addresses the challenge of transitioning pre-trained NMT models from absolute sinusoidal PEs to relative PEs. We demonstrate that parameter-efficient fine-tuning, using only a small amount of high-quality data, can successfully facilitate this transition. We find that a small amount of long-context data in a few languages is sufficient for cross-lingual length generalization.
  arXiv  Detail & Related papers  (2024-08-21T07:23:34Z)
• Base of RoPE Bounds Context Length [37.11078116104313]
  Rotary position embedding (RoPE) is a technique that encodes the position information with a rotation matrix. In this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.
  arXiv  Detail & Related papers  (2024-05-23T14:03:31Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Scaling Laws of RoPE-based Extrapolation [103.33995311915864]
  We propose textbftextitScaling Laws of RoPE-based Extrapolation to describe the relationship between the extrapolation performance and base value. We achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.
  arXiv  Detail & Related papers  (2023-10-08T15:50:36Z)
• ASR: Attention-alike Structural Re-parameterization [53.019657810468026]
  We propose a simple-yet-effective attention-alike structural re- parameterization (ASR) that allows us to achieve SRP for a given network while enjoying the effectiveness of the attention mechanism. In this paper, we conduct extensive experiments from a statistical perspective and discover an interesting phenomenon Stripe Observation, which reveals that channel attention values quickly approach some constant vectors during training.
  arXiv  Detail & Related papers  (2023-04-13T08:52:34Z)
• Adversarial Self-Attention for Language Understanding [89.265747130584]
  This paper proposes textitAdversarial Self-Attention mechanism (ASA). ASA adversarially reconstructs the Transformer attentions and facilitates model training from contaminated model structures. For fine-tuning, ASA-empowered models consistently outweigh naive models by a large margin considering both generalization and robustness.
  arXiv  Detail & Related papers  (2022-06-25T09:18:10Z)
• ES-dRNN: A Hybrid Exponential Smoothing and Dilated Recurrent Neural Network Model for Short-Term Load Forecasting [1.4502611532302039]
  Short-term load forecasting (STLF) is challenging due to complex time series (TS) This paper proposes a novel hybrid hierarchical deep learning model that deals with multiple seasonality. It combines exponential smoothing (ES) and a recurrent neural network (RNN)
  arXiv  Detail & Related papers  (2021-12-05T19:38:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.