Residual Vector-based Reconstruction as Long-Context Recall Regardless of Context Window Size

• URL: http://arxiv.org/abs/2609.12686v1
• Date: Fri, 11 Sep 2026 10:31:46 GMT
• Title: Residual Vector-based Reconstruction as Long-Context Recall Regardless of Context Window Size
• Abstract summary: Large language models (LLMs) process long contexts, including long documents and lengthy conversations.<n>LLMs face token-level memory usage that increases proportionally to input length.<n>This paper proposes a long-context recall method that maintains near-constant GPU memory usage as context length increases.
• Score: 0.6155604731137828
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) process long contexts, including long documents and lengthy conversations, but face token-level memory usage that increases proportionally to input length. Although model optimization and lossy prompt compression are widely used, these methods still fail to solve the long-context recall problem beyond pretrained and size-constrained context windows. This paper proposes a long-context recall method that maintains near-constant GPU memory usage as context length increases, without additional training. The main idea is to reconstruct facts using parameter activations in the LLM's feed-forward layers, which store residual vectors representing facts from the source document. Utilizing residual vectors allows the LLM to deterministically reconstruct query relevant facts without referencing the original document, preserving high fidelity and reducing memory usage without fine-tuning weights. Experimental results show that the proposed method enables answering single-fact questions in two-million-token story contexts where previous methods fail.

Related papers

• LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory [10.916121108457537]
  We introduce LazyMem, which deferring all memory construction to query time.<n>On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline.<n>It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline.
  arXiv  Detail & Related papers  (2026-07-17T12:49:57Z)
• ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning [78.94047086588078]
  We propose Recursive Evidence as LLM Harness for Long-Context Reasoning.<n>It is a training-free inference method for improving long-context reasoning.<n> Experiments on eight long-context datasets with 128K context length show it consistently improves evidence utilization.
  arXiv  Detail & Related papers  (2026-07-02T17:59:26Z)
• Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory [88.81430082035617]
  LLM agents require a memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history.<n>We propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity.
  arXiv  Detail & Related papers  (2026-05-19T15:05:06Z)
• Context Recycling for Long-Horizon LLM Inference [0.3655021726150367]
  Large language models (LLMs) exhibit strong capabilities in short-context reasoning but degrade in performance over long conversational horizons.<n>We introduce ContextForge, a system for context recycling that maintains task-relevant information across turns.<n>System enables efficient reuse of prior computation without relying on full context replay.
  arXiv  Detail & Related papers  (2026-05-01T04:45:08Z)
• NextMem: Towards Latent Factual Memory for LLM-based Agents [58.35585202907478]
  NextMem is a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory.<n>Extensive experiments demonstrate that NextMem achieves superior performance.
  arXiv  Detail & Related papers  (2026-02-26T14:35:27Z)
• Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning [47.87361916374891]
  We propose a framework for efficient long-context inference based on chunk-wise compression and selective memory recall.<n>The framework segments long inputs into chunks and encodes each chunk into compressed memory representations using a learned compressor.<n>It achieves up to a 2 times reduction in peak GPU memory usage and a 6 times inference speedup over MemAgent.
  arXiv  Detail & Related papers  (2026-02-09T08:33:11Z)
• Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents [33.617262543252494]
  We present ReMemR1, a memory-augmented agent with callback-enhanced memory that allows selective retrieval from the entire memory history.<n>We also propose Reinforcement Learning with Multi-Level Rewards (RLMLR), which combines final-answer rewards with dense, step-level signals that guide effective memory use.
  arXiv  Detail & Related papers  (2025-09-27T01:36:46Z)
• Needle in the Haystack for Memory Based Large Language Models [31.885539843977472]
  Current large language models (LLMs) often perform poorly on simple fact retrieval tasks. We investigate if coupling a dynamically adaptable external memory to a LLM can alleviate this problem. We demonstrate that the external memory of Larimar, which allows fast write and read of an episode of text samples, can be used at test time to handle contexts much longer than those seen during training.
  arXiv  Detail & Related papers  (2024-07-01T16:32:16Z)
• Recurrent Context Compression: Efficiently Expanding the Context Window of LLM [22.595457889113668]
  This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of Transformer-based large language models (LLMs) We validated our approach on multiple tasks, achieving a compression rate of up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and nearly 100% accuracy on a passkey retrieval task with a sequence length of 1M.
  arXiv  Detail & Related papers  (2024-06-10T08:50:59Z)
• Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs [61.40047491337793]
  We present Hierarchical cOntext MERging (HOMER), a new training-free scheme designed to overcome the limitations of large language models. HomeR uses a divide-and-conquer algorithm, dividing long inputs into manageable chunks. A token reduction technique precedes each merging, ensuring memory usage efficiency.
  arXiv  Detail & Related papers  (2024-04-16T06:34:08Z)
• Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models [30.48902594738911]
  Given a long conversation, large language models (LLMs) fail to recall past information and tend to generate inconsistent responses.<n>We propose to generate summaries/ memory using large language models (LLMs) to enhance long-term memory ability.
  arXiv  Detail & Related papers  (2023-08-29T04:59:53Z)
• Augmenting Language Models with Long-Term Memory [142.04940250657637]
  Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit. We propose a framework, Language Models Augmented with Long-Term Memory (LongMem), which enables LLMs to memorize long history.
  arXiv  Detail & Related papers  (2023-06-12T15:13:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.