Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding

• URL: http://arxiv.org/abs/2507.14849v1
• Date: Sun, 20 Jul 2025 07:43:16 GMT
• Title: Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
• Authors: Yifei Wang,
• Abstract summary: Reasoning distillation has emerged as an effective approach to enhance the reasoning capabilities of smaller language models.<n>The impact of large-scale reasoning distillation on other critical abilities, particularly in-context retrieval and reasoning, remains unexplored.
• Score: 16.50502775216771
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning distillation has emerged as an effective approach to enhance the reasoning capabilities of smaller language models. However, the impact of large-scale reasoning distillation on other critical abilities, particularly in-context retrieval and reasoning, remains unexplored. This gap in understanding is particularly significant given the increasing importance of Retrieval-Augmented Generation (RAG) systems, where efficient acquisition and utilization of contextual information are paramount for generating reliable responses. Motivated by the need to understand how the extended long-CoT process influences long-context comprehension, we conduct a comprehensive investigation using a series of open-source models distilled from Deepseek-R1, renowned for its exceptional reasoning capabilities. Our study focuses on evaluating these models' performance in extracting and integrating relevant information from extended contexts through multi-document question and answering tasks. Through rigorous experimentation, we demonstrate that distilled reasoning patterns significantly improve long-context understanding. Our analysis reveals that distillation fosters greater long-context awareness by promoting more detailed and explicit reasoning processes during context analysis and information parsing. This advancement effectively mitigates the persistent "lost in the middle" issue that has hindered long-context models.

Related papers

• Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Models [21.579319926212296]
  Large Language Models (LLMs) have emerged as powerful tools for generating coherent text, understanding context, and performing reasoning tasks.<n>They struggle with temporal reasoning, which requires processing time-related information such as event sequencing, durations, and inter-temporal relationships.<n>We introduce TISER, a novel framework that enhances the temporal reasoning abilities of LLMs through a multi-stage process that combines timeline construction with iterative self-reflection.
  arXiv  Detail & Related papers  (2025-04-07T16:51:45Z)
• Improving Multilingual Retrieval-Augmented Language Models through Dialectic Reasoning Argumentations [65.11348389219887]
  We introduce Dialectic-RAG (DRAG), a modular approach that evaluates retrieved information by comparing, contrasting, and resolving conflicting perspectives.<n>We show the impact of our framework both as an in-context learning strategy and for constructing demonstrations to instruct smaller models.
  arXiv  Detail & Related papers  (2025-04-07T06:55:15Z)
• Reasoning on Multiple Needles In A Haystack [9.765859280987053]
  We tackle the memory-based answering problem by filtering out direct-answer questions.<n>We build on this insight to introduce a reflection mechanism for multi-round extension.<n>We train a model using the generated iterative thinking process, which helps mitigate the performance degradation.
  arXiv  Detail & Related papers  (2025-04-05T11:58:08Z)
• A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond [88.5807076505261]
  Large Reasoning Models (LRMs) have demonstrated strong performance gains by scaling up the length of Chain-of-Thought (CoT) reasoning during inference.<n>A growing concern lies in their tendency to produce excessively long reasoning traces.<n>This inefficiency introduces significant challenges for training, inference, and real-world deployment.
  arXiv  Detail & Related papers  (2025-03-27T15:36:30Z)
• Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval [33.84832445715185]
  Large Language Models (LLMs) often exhibit substantially shorter effective context lengths than their claimed capacities.<n>We propose a novel training-free algorithm, Attrieval, which leverages attention weights to retrieve relevant facts from the long context.<n>Our results demonstrate that Attrieval enhances long-context reasoning capability notably on both synthetic and real-world QA datasets.
  arXiv  Detail & Related papers  (2025-03-12T20:34:14Z)
• LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data [19.79929012055293]
  LongFaith is a novel pipeline for synthesizing faithful long-context reasoning instruction datasets.<n>By integrating ground truth and citation-based reasoning prompts, we eliminate distractions and improve the accuracy of reasoning chains.
  arXiv  Detail & Related papers  (2025-02-18T06:40:23Z)
• Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models [62.698520962933195]
  Large Vision-Language Models (LVLMs) excel in cross-model tasks but experience performance declines in long-context reasoning.<n>We propose a novel training-free context pruning method that selectively removes less critical textual information.
  arXiv  Detail & Related papers  (2024-10-25T17:59:09Z)
• DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels [86.93099925711388]
  We propose textbfDetectiveQA, a dataset specifically designed for narrative reasoning within long contexts.<n>We leverage detective novels, averaging over 100k tokens, to create a dataset containing 1200 human-annotated questions in both Chinese and English.
  arXiv  Detail & Related papers  (2024-09-04T06:28:22Z)
• QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory [66.01597794579568]
  We introduce information bottleneck theory (IB) to model the problem.<n>We propose a cross-attention-based approach to approximate mutual information in IB.<n>Our method achieves a 25% increase in compression rate compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2024-08-20T02:44:45Z)
• Unlocking Temporal Question Answering for Large Language Models with Tailor-Made Reasoning Logic [84.59255070520673]
  Large language models (LLMs) face a challenge when engaging in temporal reasoning. We propose TempLogic, a novel framework designed specifically for temporal question-answering tasks.
  arXiv  Detail & Related papers  (2023-05-24T10:57:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.