Flora: Effortless Context Construction to Arbitrary Length and Scale

• URL: http://arxiv.org/abs/2507.19786v1
• Date: Sat, 26 Jul 2025 04:21:21 GMT
• Title: Flora: Effortless Context Construction to Arbitrary Length and Scale
• Authors: Tianxiang Chen, Zhentao Tan, Xiaofan Bo, Yue Wu, Tao Gong, Qi Chu, Jieping Ye, Nenghai Yu,
• Abstract summary: We introduce Flora, an effortless (human/LLM-free) long-context construction strategy.<n>Experiments on Llama3-8B-Instruct and QwQ-32B show that Flora excel in three long-context benchmarks while maintaining strong performances in short-context tasks.
• Score: 71.12886910497284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effectively handling long contexts is challenging for Large Language Models (LLMs) due to the rarity of long texts, high computational demands, and substantial forgetting of short-context abilities. Recent approaches have attempted to construct long contexts for instruction tuning, but these methods often require LLMs or human interventions, which are both costly and limited in length and diversity. Also, the drop in short-context performances of present long-context LLMs remains significant. In this paper, we introduce Flora, an effortless (human/LLM-free) long-context construction strategy. Flora can markedly enhance the long-context performance of LLMs by arbitrarily assembling short instructions based on categories and instructing LLMs to generate responses based on long-context meta-instructions. This enables Flora to produce contexts of arbitrary length and scale with rich diversity, while only slightly compromising short-context performance. Experiments on Llama3-8B-Instruct and QwQ-32B show that LLMs enhanced by Flora excel in three long-context benchmarks while maintaining strong performances in short-context tasks. Our data-construction code is available at \href{https://github.com/txchen-USTC/Flora}{https://github.com/txchen-USTC/Flora}.

Related papers

• LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions [28.002824369635768]
  LongMagpie is a framework that automatically generates large-scale long-context instruction data.<n>We show that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks.
  arXiv  Detail & Related papers  (2025-05-22T04:05:02Z)
• Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning [103.65680870130839]
  We investigate how to design instruction data for the post-training phase of a long context pre-trained model.<n>Our controlled study reveals that models instruction-tuned on short contexts can effectively generalize to longer ones.<n>Based on these findings, we propose context synthesis, a novel data synthesis framework.
  arXiv  Detail & Related papers  (2025-02-21T17:02:40Z)
• LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization [49.37607974207405]
  LongPO harnesses short-to-long preference data to transfer short-context capabilities to long-context tasks.<n>LongPO fully retains short-context performance and largely outperforms naive SFT and DPO in both long- and short-context tasks.
  arXiv  Detail & Related papers  (2025-02-19T17:59:03Z)
• LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning [35.31849814789343]
  This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling.<n>LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation.<n>The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-12-18T09:04:55Z)
• What is Wrong with Perplexity for Long-context Language Modeling? [71.34933096461124]
  Long-context inputs are crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning.<n>Perplexity (PPL) has proven unreliable for assessing long-context capabilities.<n>We propose bfLongPPL, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them.
  arXiv  Detail & Related papers  (2024-10-31T09:39:28Z)
• LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models [61.12177317970258]
  LongSkywork is a long-context Large Language Model capable of processing up to 200,000 tokens. We develop two novel methods for creating synthetic data. LongSkywork achieves outstanding performance on a variety of long-context benchmarks.
  arXiv  Detail & Related papers  (2024-06-02T03:34:41Z)
• Long Context Alignment with Short Instructions and Synthesized Positions [56.1267385315404]
  This paper introduces Step-Skipping Alignment (SkipAlign) It is a new technique designed to enhance the long-context capabilities of Large Language Models (LLMs) With a careful selection of the base model and alignment datasets, SkipAlign with only 6B parameters achieves it's best performance and comparable with strong baselines like GPT-3.5-Turbo-16K on LongBench.
  arXiv  Detail & Related papers  (2024-05-07T01:56:22Z)
• LooGLE: Can Long-Context Language Models Understand Long Contexts? [46.143956498529796]
  LooGLE is a benchmark for large language models' long context understanding. It features relatively new documents post-2022, with over 24,000 tokens per document and 6,000 newly generated questions spanning diverse domains. The evaluation of eight state-of-the-art LLMs on LooGLE revealed key findings.
  arXiv  Detail & Related papers  (2023-11-08T01:45:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.