Can an LLM Induce a Graph? Investigating Memory Drift and Context Length

• URL: http://arxiv.org/abs/2510.03611v1
• Date: Sat, 04 Oct 2025 01:56:07 GMT
• Title: Can an LLM Induce a Graph? Investigating Memory Drift and Context Length
• Authors: Raquib Bin Yousuf, Aadyant Khatri, Shengzhe Xu, Mandar Sharma, Naren Ramakrishnan,
• Abstract summary: Recently proposed evaluation benchmarks aim to characterize the effective context length and the forgetting tendencies of large language models (LLMs)<n>We argue for evaluating these models on more complex reasoning tasks that require them to induce structured relational knowledge from the text.<n>Our findings reveal that LLMs begin to exhibit memory drift and contextual forgetting at much shorter effective lengths when tasked with this form of relational reasoning.
• Score: 11.214847796972705
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently proposed evaluation benchmarks aim to characterize the effective context length and the forgetting tendencies of large language models (LLMs). However, these benchmarks often rely on simplistic 'needle in a haystack' retrieval or continuation tasks that may not accurately reflect the performance of these models in information-dense scenarios. Thus, rather than simple next token prediction, we argue for evaluating these models on more complex reasoning tasks that requires them to induce structured relational knowledge from the text - such as graphs from potentially noisy natural language content. While the input text can be viewed as generated in terms of a graph, its structure is not made explicit and connections must be induced from distributed textual cues, separated by long contexts and interspersed with irrelevant information. Our findings reveal that LLMs begin to exhibit memory drift and contextual forgetting at much shorter effective lengths when tasked with this form of relational reasoning, compared to what existing benchmarks suggest. With these findings, we offer recommendations for the optimal use of popular LLMs for complex reasoning tasks. We further show that even models specialized for reasoning, such as OpenAI o1, remain vulnerable to early memory drift in these settings. These results point to significant limitations in the models' ability to abstract structured knowledge from unstructured input and highlight the need for architectural adaptations to improve long-range reasoning.

Related papers

• A Stepwise-Enhanced Reasoning Framework for Large Language Models Based on External Subgraph Generation [12.27505872942231]
  Large Language Models (LLMs) have achieved strong performance across a wide range of natural language processing tasks.<n>We propose a stepwise reasoning enhancement framework for LLMs based on external subgraph generation.<n>By performing reasoning in a step by step manner over structured subgraphs, SGR reduces the influence of noisy information and improves reasoning accuracy.
  arXiv  Detail & Related papers  (2025-12-29T10:35:53Z)
• Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles [81.89404347890662]
  SciTrek is a novel question-answering benchmark designed to evaluate the long-context reasoning capabilities of large language models (LLMs) using scientific articles.<n>Our analysis reveals systematic shortcomings in models' abilities to perform basic numerical operations and accurately locate specific information in long contexts.
  arXiv  Detail & Related papers  (2025-09-25T11:36:09Z)
• GRIL: Knowledge Graph Retrieval-Integrated Learning with Large Language Models [59.72897499248909]
  We propose a novel graph retriever trained end-to-end with Large Language Models (LLMs)<n>Within the extracted subgraph, structural knowledge and semantic features are encoded via soft tokens and the verbalized graph, respectively, which are infused into the LLM together.<n>Our approach consistently achieves state-of-the-art performance, validating the strength of joint graph-LLM optimization for complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-09-20T02:38:00Z)
• Towards Explainable Temporal Reasoning in Large Language Models: A Structure-Aware Generative Framework [18.770512964705176]
  We introduce a benchmark covering a wide range of temporal granularities to systematically evaluate large language models' capabilities in explainable temporal reasoning.<n>We propose GETER, a novel structure-aware generative framework that integrates Graph structures with text for Explainable TEmporal Reasoning.<n> GETER achieves state-of-the-art performance while also demonstrating its effectiveness as well as strong generalization capabilities.
  arXiv  Detail & Related papers  (2025-05-21T08:20:35Z)
• Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM [11.181783720439563]
  Large Language Models (LLMs) display sophisticated reasoning abilities via extended Chain-of-Thought (CoT) generation.<n>RLMs often demonstrate counterintuitive and unstable behaviors, such as performance degradation under few-shot prompting.<n>We introduce a unified graph-based analytical framework for better modeling the reasoning processes of RLMs.
  arXiv  Detail & Related papers  (2025-05-20T03:54:57Z)
• Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning [103.65680870130839]
  We investigate how to design instruction data for the post-training phase of a long context pre-trained model.<n>Our controlled study reveals that models instruction-tuned on short contexts can effectively generalize to longer ones.<n>Based on these findings, we propose context synthesis, a novel data synthesis framework.
  arXiv  Detail & Related papers  (2025-02-21T17:02:40Z)
• Navigating Semantic Relations: Challenges for Language Models in Abstract Common-Sense Reasoning [5.4141465747474475]
  Large language models (LLMs) have achieved remarkable performance in generating human-like text and solving problems of moderate complexity.<n>We systematically evaluate abstract common-sense reasoning in LLMs using the ConceptNet knowledge graph.
  arXiv  Detail & Related papers  (2025-02-19T20:20:24Z)
• NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities [51.07379913779232]
  NeedleBench is a framework for assessing retrieval and reasoning performance in long-context tasks.<n>It embeds key data points at varying depths to rigorously test model capabilities.<n>Our experiments reveal that reasoning models like Deep-R1 and OpenAI's o3 struggle with continuous retrieval and reasoning in information-dense scenarios.
  arXiv  Detail & Related papers  (2024-07-16T17:59:06Z)
• Optimizing Language Model's Reasoning Abilities with Weak Supervision [48.60598455782159]
  We present textscPuzzleBen, a weakly supervised benchmark that comprises 25,147 complex questions, answers, and human-generated rationales. A unique aspect of our dataset is the inclusion of 10,000 unannotated questions, enabling us to explore utilizing fewer supersized data to boost LLMs' inference capabilities.
  arXiv  Detail & Related papers  (2024-05-07T07:39:15Z)
• Learning to Reduce: Optimal Representations of Structured Data in Prompting Large Language Models [42.16047343029512]
  Large Language Models (LLMs) have been widely used as general-purpose AI agents. We propose a framework, Learning to Reduce, that fine-tunes a language model to generate a reduced version of an input context. We show that our model achieves comparable accuracies in selecting the relevant evidence from an input context.
  arXiv  Detail & Related papers  (2024-02-22T00:41:23Z)
• Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Construction [57.854498238624366]
  We propose a retrieval-augmented approach, which retrieves schema-aware Reference As Prompt (RAP) for data-efficient knowledge graph construction. RAP can dynamically leverage schema and knowledge inherited from human-annotated and weak-supervised data as a prompt for each sample.
  arXiv  Detail & Related papers  (2022-10-19T16:40:28Z)
• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning [85.33459673197149]
  We introduce a new Reading dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. In this paper, we propose to identify biased data points and separate them into EASY set and the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.
  arXiv  Detail & Related papers  (2020-02-11T11:54:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.