LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization

• URL: http://arxiv.org/abs/2502.13922v2
• Date: Thu, 20 Feb 2025 09:16:36 GMT
• Title: LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
• Authors: Guanzheng Chen, Xin Li, Michael Qizhe Shieh, Lidong Bing,
• Abstract summary: LongPO harnesses short-to-long preference data to transfer short-context capabilities to long-context tasks. LongPO fully retains short-context performance and largely outperforms naive SFT and DPO in both long- and short-context tasks.
• Score: 49.37607974207405
• License:
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities through pretraining and alignment. However, superior short-context LLMs may underperform in long-context scenarios due to insufficient long-context alignment. This alignment process remains challenging due to the impracticality of human annotation for extended contexts and the difficulty in balancing short- and long-context performance. To address these challenges, we introduce LongPO, that enables short-context LLMs to self-evolve to excel on long-context tasks by internally transferring short-context capabilities. LongPO harnesses LLMs to learn from self-generated short-to-long preference data, comprising paired responses generated for identical instructions with long-context inputs and their compressed short-context counterparts, respectively. This preference reveals capabilities and potentials of LLMs cultivated during short-context alignment that may be diminished in under-aligned long-context scenarios. Additionally, LongPO incorporates a short-to-long KL constraint to mitigate short-context performance decline during long-context alignment. When applied to Mistral-7B-Instruct-v0.2 from 128K to 512K context lengths, LongPO fully retains short-context performance and largely outperforms naive SFT and DPO in both long- and short-context tasks. Specifically, LongPO-trained models can achieve results on long-context benchmarks comparable to, or even surpassing, those of superior LLMs (e.g., GPT-4-128K) that involve extensive long-context annotation and larger parameter scales. Our code is available at https://github.com/DAMO-NLP-SG/LongPO.

Related papers

• LIFT: Improving Long Context Understanding of Large Language Models through Long Input Fine-Tuning [45.30182393918228]
  Long Input Fine-Tuning (LIFT) is a novel framework for long-context modeling. LIFT dynamically adapts model parameters based on the long input. Gated Memory is a specialized attention adapter that automatically balances long input memorization and ICL.
  arXiv  Detail & Related papers  (2025-02-20T15:32:24Z)
• LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning [35.31849814789343]
  This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling. LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation. The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-12-18T09:04:55Z)
• What is Wrong with Perplexity for Long-context Language Modeling? [71.34933096461124]
  Long-context inputs are crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. Perplexity (PPL) has proven unreliable for assessing long-context capabilities. We propose bfLongPPL, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them.
  arXiv  Detail & Related papers  (2024-10-31T09:39:28Z)
• LongReward: Improving Long-context Large Language Models with AI Feedback [54.3321542678909]
  LongReward is a novel method that provides rewards for long-context model responses from four human-valued dimensions. Our experiments indicate that LongReward not only significantly improves models' long-context performance but also enhances their ability to follow short instructions.
  arXiv  Detail & Related papers  (2024-10-28T17:50:42Z)
• LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models [61.12177317970258]
  LongSkywork is a long-context Large Language Model capable of processing up to 200,000 tokens. We develop two novel methods for creating synthetic data. LongSkywork achieves outstanding performance on a variety of long-context benchmarks.
  arXiv  Detail & Related papers  (2024-06-02T03:34:41Z)
• Long Context Alignment with Short Instructions and Synthesized Positions [56.1267385315404]
  This paper introduces Step-Skipping Alignment (SkipAlign) It is a new technique designed to enhance the long-context capabilities of Large Language Models (LLMs) With a careful selection of the base model and alignment datasets, SkipAlign with only 6B parameters achieves it's best performance and comparable with strong baselines like GPT-3.5-Turbo-16K on LongBench.
  arXiv  Detail & Related papers  (2024-05-07T01:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.