LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

• URL: http://arxiv.org/abs/2607.06160v1
• Date: Tue, 07 Jul 2026 11:35:48 GMT
• Title: LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
• Abstract summary: LongCrafter is a structured synthesis framework that couples a hierarchical task taxonomy with an evidence-grounded pipeline.<n>LongCrafter constructs task-aligned long contexts, decomposes them into explicit evidence graphs that model cross-paragraph dependencies, and generates instruction--response pairs strictly grounded in the located evidence spans.<n>Models fine-tuned on LongCrafter data outperform all SFT baselines and even the official post-trained models.
• Score: 25.64387650835006
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision. We propose \textbf{LongCrafter}, a structured synthesis framework that couples a hierarchical task taxonomy with an evidence-grounded pipeline. The taxonomy organizes long-context understanding into local/shallow and global/deep levels and yields 32 fine-grained task types that serve as a global generative prior. Guided by this taxonomy, LongCrafter constructs task-aligned long contexts, decomposes them into explicit evidence graphs that model cross-paragraph dependencies, and generates instruction--response pairs strictly grounded in the located evidence spans, ensuring both controllable difficulty and faithful, traceable reasoning. Models fine-tuned on LongCrafter data outperform all SFT baselines and even the official post-trained models on LongBench, LongBench~v2, and LooGLE across both Qwen2.5-7B and LLaMA-3.1-8B, with the largest gains on high-difficulty tasks. Further analysis shows that LongCrafter data is more diverse and better spread across difficulty levels, and that the trained models locate evidence robustly regardless of position, effectively mitigating the ``lost in the middle'' problem.

Related papers

• Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench [6.490490294075993]
  We propose PredicateLongBench as a benchmark for evaluating large language models.<n>It asks models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints.<n>We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes.
  arXiv  Detail & Related papers  (2026-07-09T09:28:06Z)
• IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking [50.381430711141995]
  We introduce the Interleaved Structural Chain-of-Thought (IS-CoT) framework.<n>IS-CoT embeds a dynamic Plan-Write-Reflect cycle into the generation process.<n>We construct a high-quality dataset of interleaved reasoning traces via a multi-teacher pipeline and train IS-Writer-8B.
  arXiv  Detail & Related papers  (2026-06-08T16:31:00Z)
• DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding [63.257540233507626]
  We propose a paradigm that requires the model to execute a structured Analysis, Localization and Reasoning workflow.<n>We show it robustly generalizes from short-page training to ultra-long documents and is naturally synergistic with visual Retrieval-Augmented Generation systems.
  arXiv  Detail & Related papers  (2026-04-14T14:39:26Z)
• WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale [86.25450054683172]
  WildLong extracts meta-information from real user queries to produce scalable data.<n>It supports multi-document reasoning, such as cross-document comparison and aggregation.<n>It surpasses existing open-source long-context-optimized models across benchmarks.
  arXiv  Detail & Related papers  (2025-02-23T18:59:09Z)
• Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning [103.65680870130839]
  We investigate how to design instruction data for the post-training phase of a long context pre-trained model.<n>Our controlled study reveals that models instruction-tuned on short contexts can effectively generalize to longer ones.<n>Based on these findings, we propose context synthesis, a novel data synthesis framework.
  arXiv  Detail & Related papers  (2025-02-21T17:02:40Z)
• LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data [19.79929012055293]
  LongFaith is a novel pipeline for synthesizing faithful long-context reasoning instruction datasets.<n>By integrating ground truth and citation-based reasoning prompts, we eliminate distractions and improve the accuracy of reasoning chains.
  arXiv  Detail & Related papers  (2025-02-18T06:40:23Z)
• GATEAU: Selecting Influential Samples for Long Context Alignment [59.579128690086385]
  GATEAU identifies influential samples enriched with long-range dependency relations.<n>Model trained on selected samples exhibits better instruction-following and long-context understanding capabilities.
  arXiv  Detail & Related papers  (2024-10-21T04:30:53Z)
• Long Context Alignment with Short Instructions and Synthesized Positions [56.1267385315404]
  This paper introduces Step-Skipping Alignment (SkipAlign) It is a new technique designed to enhance the long-context capabilities of Large Language Models (LLMs) With a careful selection of the base model and alignment datasets, SkipAlign with only 6B parameters achieves it's best performance and comparable with strong baselines like GPT-3.5-Turbo-16K on LongBench.
  arXiv  Detail & Related papers  (2024-05-07T01:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.