A Decomposition Perspective to Long-context Reasoning for LLMs

• URL: http://arxiv.org/abs/2604.07981v1
• Date: Thu, 09 Apr 2026 08:51:47 GMT
• Title: A Decomposition Perspective to Long-context Reasoning for LLMs
• Abstract summary: Long-context reasoning is essential for complex real-world applications.<n>This paper decomposes long-context reasoning into a set of fundamental atomic skills.<n>We employ reinforcement learning to sharpen the model's atomic skills.
• Score: 33.33617233628542
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-context reasoning is essential for complex real-world applications, yet remains a significant challenge for Large Language Models (LLMs). Despite the rapid evolution in long-context reasoning, current research often overlooks the internal complexity of the long-context reasoning task itself. In this paper, we move beyond this holistic view and decompose long-context reasoning into a set of fundamental atomic skills, and we then automatically synthesize a suite of pseudo datasets, each explicitly targeting a specific atomic skill. Our empirical analysis confirms that proficiency in these atomic skills is strongly correlated with general long-text reasoning performance. Building on this insight, we employ reinforcement learning on these pseudo datasets to sharpen the model's atomic skills, in the hope of boosting its general long-context reasoning ability. Extensive experiments across multiple benchmarks demonstrate the effectiveness of our approach: it outperforms a strong baseline by an average margin of 7.7\% (improving from 46.3\% to 54.0\%) across Loogle, Loong, LongBench-v2, BrowscompLong, Ruler-qa2, and MRCR.

Related papers

• LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning [54.6434454250226]
  LongAct is a strategy that shifts from uniform to saliency-guided sparse updates.<n>By selectively updating only the weights associated with these significant activations, LongAct achieves an approximate 8% improvement on LongBench v2.
  arXiv  Detail & Related papers  (2026-04-16T12:06:59Z)
• Probing How Scalable Table Data Enhances General Long-Context Reasoning [18.383487310920597]
  We find that structured table data with periodic structures shows strong potential for long-context reasoning.<n>We propose a simple yet scalable pipeline(TableLong) for synthesizing high-quality, diverse, and verifiable structured table data.
  arXiv  Detail & Related papers  (2026-03-23T09:05:46Z)
• Context-Length Robustness in Question Answering Models: A Comparative Empirical Study [0.0]
  We present an empirical study of context-length robustness in large language models using two benchmarks: SQuAD and HotpotQA.<n>We evaluate model accuracy as a function of total context length by systematically increasing the amount of irrelevant context while preserving the answer-bearing signal.<n>Our results show a consistent degradation in performance as context length increases, with substantially larger drops observed on multi-hop reasoning tasks compared to single-span extraction tasks.
  arXiv  Detail & Related papers  (2026-03-16T17:14:05Z)
• LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards [57.993003392037174]
  LongR is a framework that enhances long-context performance by integrating a dynamic "Think-and-Read" mechanism.<n>LongR consistently enhances performance across diverse RL algorithms.
  arXiv  Detail & Related papers  (2026-02-05T15:26:47Z)
• A Controllable Examination for Long-Context Language Models [62.845852724511964]
  This study introduces $textbfLongBioBench, a benchmark for evaluating long-context language models.<n>We show that most models still exhibit deficiencies in semantic understanding and elementary reasoning over retrieved results.<n>Our further analysis indicates some design choices employed by existing synthetic benchmarks, such as contextual non-coherence.
  arXiv  Detail & Related papers  (2025-06-03T14:23:06Z)
• Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning [23.301029291780317]
  We examine whether enhancing a model's long-context ability beforeSupervised Fine-Tuning (SFT) leads to improved reasoning performance.<n>Our results reveal a consistent trend: models with stronger long-context capacity achieve significantly higher accuracy on reasoning benchmarks after SFT.<n>These findings suggest that long-context modeling is not just essential for processing lengthy inputs, but also serves as a critical foundation for reasoning.
  arXiv  Detail & Related papers  (2025-05-22T22:09:47Z)
• FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models [59.171510592986735]
  We propose FactReasoner, a new factuality assessor that relies on probabilistic reasoning to assess the factuality of a long-form generated response.<n>Our experiments on labeled and unlabeled benchmark datasets demonstrate clearly that FactReasoner improves considerably over state-of-the-art prompt-based approaches.
  arXiv  Detail & Related papers  (2025-02-25T19:01:48Z)
• LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data [19.79929012055293]
  LongFaith is a novel pipeline for synthesizing faithful long-context reasoning instruction datasets.<n>By integrating ground truth and citation-based reasoning prompts, we eliminate distractions and improve the accuracy of reasoning chains.
  arXiv  Detail & Related papers  (2025-02-18T06:40:23Z)
• NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities [51.07379913779232]
  NeedleBench is a framework for assessing retrieval and reasoning performance in long-context tasks.<n>It embeds key data points at varying depths to rigorously test model capabilities.<n>Our experiments reveal that reasoning models like Deep-R1 and OpenAI's o3 struggle with continuous retrieval and reasoning in information-dense scenarios.
  arXiv  Detail & Related papers  (2024-07-16T17:59:06Z)
• Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA [71.04146366608904]
  Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. We propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning.
  arXiv  Detail & Related papers  (2024-06-25T09:42:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.