LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models

• URL: http://arxiv.org/abs/2406.00605v1
• Date: Sun, 2 Jun 2024 03:34:41 GMT
• Title: LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
• Authors: Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, Yahui Zhou,
• Abstract summary: LongSkywork is a long-context Large Language Model capable of processing up to 200,000 tokens. We develop two novel methods for creating synthetic data. LongSkywork achieves outstanding performance on a variety of long-context benchmarks.
• Score: 61.12177317970258
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce LongSkywork, a long-context Large Language Model (LLM) capable of processing up to 200,000 tokens. We provide a training recipe for efficiently extending context length of LLMs. We identify that the critical element in enhancing long-context processing capability is to incorporate a long-context SFT stage following the standard SFT stage. A mere 200 iterations can convert the standard SFT model into a long-context model. To reduce the effort in collecting and annotating data for long-context language modeling, we develop two novel methods for creating synthetic data. These methods are applied during the continual pretraining phase as well as the Supervised Fine-Tuning (SFT) phase, greatly enhancing the training efficiency of our long-context LLMs. Our findings suggest that synthetic long-context SFT data can surpass the performance of data curated by humans to some extent. LongSkywork achieves outstanding performance on a variety of long-context benchmarks. In the Needle test, a benchmark for long-context information retrieval, our models achieved perfect accuracy across multiple context spans. Moreover, in realistic application scenarios, LongSkywork-13B demonstrates performance on par with Claude2.1, the leading long-context model, underscoring the effectiveness of our proposed methods.

Related papers

• Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation [15.975325252309554]
  We introduce a novel post-training synthetic data generation strategy designed to efficiently extend the context window of Large Language Models. Our approach scalably extends to arbitrarily long context lengths, unconstrained by the length of available real-world data. We demonstrate that our model, with a context length of up to 1M tokens, performs well on the RULER benchmark and InfiniteBench.
  arXiv  Detail & Related papers  (2025-04-17T04:46:57Z)
• From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models [54.44375226381814]
  Long-context capabilities are essential for a wide range of applications, including document and video understanding, in-context learning, and inference-time scaling. We introduce a efficient training recipe for building ultra-long context LLMs from aligned instruct model, pushing the boundaries of context lengths from 128K to 1M, 2M, and 4M tokens. Our approach achieves state-of-the-art performance across a diverse set of long-context benchmarks.
  arXiv  Detail & Related papers  (2025-04-08T16:58:58Z)
• LongAttn: Selecting Long-context Training Data via Token-level Attention [16.30530770590871]
  LongAttn is a token-level framework to measure the long-range dependencies for the data. We filter LongABC-32K from open-source long-context datasets (ArXiv, Book, and Code)
  arXiv  Detail & Related papers  (2025-02-24T05:51:53Z)
• Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning [103.65680870130839]
  We investigate how to design instruction data for the post-training phase of a long context pre-trained model. Our controlled study reveals that models instruction-tuned on short contexts can effectively generalize to longer ones. Based on these findings, we propose context synthesis, a novel data synthesis framework.
  arXiv  Detail & Related papers  (2025-02-21T17:02:40Z)
• LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning [35.31849814789343]
  This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling. LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation. The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-12-18T09:04:55Z)
• LongReward: Improving Long-context Large Language Models with AI Feedback [54.3321542678909]
  LongReward is a novel method that provides rewards for long-context model responses from four human-valued dimensions. Our experiments indicate that LongReward not only significantly improves models' long-context performance but also enhances their ability to follow short instructions.
  arXiv  Detail & Related papers  (2024-10-28T17:50:42Z)
• How to Train Long-Context Language Models (Effectively) [75.5418485597276]
  We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. ProLong-8B, which is from Llama-3 and trained on 40B tokens, demonstrates state-of-the-art long-context performance among similarly sized models at a length of 128K.
  arXiv  Detail & Related papers  (2024-10-03T16:46:52Z)
• Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models [21.90388980448712]
  Training models to handle long contexts presents significant challenges. We introduce Untie the Knots (textbfUtK), a novel data augmentation strategy employed during the continue pre-training phase. We conduct extensive experiments on models with 7B and 72B parameters, trained on 20 billion tokens, demonstrating that UtK achieves 75% and 84.5% accurracy on RULER at 128K context length.
  arXiv  Detail & Related papers  (2024-09-07T09:28:55Z)
• Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models [13.091271774417867]
  Long-context modeling capabilities are important for large language models (LLMs) in various applications. We propose a data mining framework textbfProLong that can assign each training sample with a long dependency score. Comprehensive experiments on multiple benchmarks indicate that ProLong effectively identifies documents that carry long dependencies.
  arXiv  Detail & Related papers  (2024-05-28T07:36:56Z)
• Long Context Alignment with Short Instructions and Synthesized Positions [56.1267385315404]
  This paper introduces Step-Skipping Alignment (SkipAlign) It is a new technique designed to enhance the long-context capabilities of Large Language Models (LLMs) With a careful selection of the base model and alignment datasets, SkipAlign with only 6B parameters achieves it's best performance and comparable with strong baselines like GPT-3.5-Turbo-16K on LongBench.
  arXiv  Detail & Related papers  (2024-05-07T01:56:22Z)
• E^2-LLM: Efficient and Extreme Length Extension of Large Language Models [74.1254067728251]
  We propose an Efficient and Extreme length extension method for Large Language Models, called E 2 -LLM, with only one training procedure and dramatically reduced cost. Comprehensive experimental results on multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on challenging long-context tasks.
  arXiv  Detail & Related papers  (2024-01-13T02:11:20Z)
• Effective Long-Context Scaling of Foundation Models [90.57254298730923]
  We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2.
  arXiv  Detail & Related papers  (2023-09-27T21:41:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.