WhenLoss: Diagnosing Write and Retrieval Bottlenecks in Long-Context Memory Systems

• URL: http://arxiv.org/abs/2605.24579v1
• Date: Sat, 23 May 2026 13:43:14 GMT
• Title: WhenLoss: Diagnosing Write and Retrieval Bottlenecks in Long-Context Memory Systems
• Abstract summary: Expected Predictive Compression achieves the highest scores among all systems.<n>These results suggest that, on this benchmark and evaluation setup, improving what the write stage preserves is a key avenue for performance gains.
• Score: 2.605344682123109
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-context memory systems often fail under fixed budgets, but end-to-end evaluation does not reveal whether evidence was discarded during compression or preserved but never retrieved. We introduce a four-condition diagnostic protocol that evaluates a fixed reader under truncated full context (TFC), oracle evidence (OE), complete stored memory (CSM), and retrieved memory (RM). Under this fixed-budget LongMemEval setup, write-side gaps exceed retrieval-side gaps for most tested baselines, with four of six baselines robustly write-dominant under our default diagnosis margin. Motivated by this diagnosis, we propose Expected Predictive Compression (EPC), which moves the key decision--what information to retain--to write time by using an LLM to anticipate likely future questions and preserve the minimal supporting evidence under the token budget, while leaving retrieval unchanged at question time. Across all 500 LongMemEval questions with three readers (GPT-5.2, Claude Sonnet 4, Gemini 2.5 Pro), EPC achieves the highest CSM scores among all systems (0.49 vs. 0.44 for Summary (LLM), the strongest baseline), reducing Delta_write to 0.04 while leaving Delta_retr comparable to other LLM-based systems. These results suggest that, on this benchmark and evaluation setup, improving what the write stage preserves is a key avenue for performance gains in the tested systems.

Related papers

• Continual Graph Memory for Adaptive Recommendation under Intent Drift [7.140905344515738]
  This paper studies adaptive recommendation under intent drift, where feedback from each recommendation outcome can reveal whether the relational evidence used for ranking is useful, missing, or misleading.<n>This paper proposes CGM-Rec, a continual graph memory framework for adaptive recommendation.
  arXiv  Detail & Related papers  (2026-09-04T02:32:20Z)
• MemGuard: Persisting Verifier Signals for LLM-Agent Memory Governance [21.021172368088504]
  LLM agents are moving from single-prompt use to long task streams in which reusable memory becomes a core capability.<n>MemGuard treats verifier output not as a one-shot filter, but as persistent lifecycle metadata.<n>MemGuard achieves the best success metric and lowest average steps in all 16 backbone-benchmark settings.
  arXiv  Detail & Related papers  (2026-08-22T09:25:23Z)
• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents [97.19033877543522]
  We present a controlled harness evaluation of memory substrates for memory-augmented agents.<n>We instrument 26 performance and efficiency metrics under a unified harness.<n>These findings motivate substrate routing as a necessary component of adaptive agent memory systems.
  arXiv  Detail & Related papers  (2026-08-15T03:29:48Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• Memory-Managed Long-Context Attention: A Preliminary Study of Editable Request-Local Memory [0.5371337604556311]
  We study memory-managed long-context attention, a research route that separates a fast recurrent or sparse backbone from explicit editable request-local memory slots.<n>Across structured synthetic tasks, token/chunk/sequence bridges, generated natural language, and local frozen-model diagnostics, pure fixed-state or pure sparse methods fail some overwrite, version, anti-pollution, or no-write-signal cases.
  arXiv  Detail & Related papers  (2026-06-27T11:38:43Z)
• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention [77.12062766962815]
  Lookahead Sparse Attention (LSA) is powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture.<n>We instantiate this architecture via a backbone-free decoupled training strategy.<n>We show that FM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline.
  arXiv  Detail & Related papers  (2026-06-08T06:25:54Z)
• MemFail: Stress-Testing Failure Modes of LLM Memory Systems [69.80981631587501]
  Large language model (LLM) agents increasingly rely on external memory systems to remain consistent across long-horizon interactions.<n>Existing benchmarks report aggregate question-answering accuracy and treat memory systems as black boxes.<n>We introduce MemFail, a diagnostic benchmark that isolates the failure modes of modern LLM memory systems.
  arXiv  Detail & Related papers  (2026-05-26T08:03:55Z)
• MemGym: a Long-Horizon Memory Environment for LLM Agents [69.79226770543049]
  We present MemGym, a benchmark for agentic memory that unifies existing agent gyms and in-house memory-grounded pipelines.<n>MemGym reports memory-isolated scores that decouple memory performance from reasoning, retrieval, and tool-use ability.<n>Our synthetic pipelines for MEMGYM-CODEQA and MEMGYM-DR are length-controllable, ablation-verified at every stage, and tightly aligned with downstream scenarios.
  arXiv  Detail & Related papers  (2026-05-20T07:25:33Z)
• Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory [50.857546269660276]
  We introduce SeqMem-Eval, a diagnostic evaluation framework for sequentially evolving large language models (LLMs) memory.<n>Rather than focusing only on final performance, SeqMem-Eval evaluates how memory states evolve, generalize, consolidate experience, and retain useful information during sequential inference.
  arXiv  Detail & Related papers  (2026-05-14T20:15:22Z)
• LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues [80.29362825271768]
  LongMemEval-V2 is a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments.<n>LME-V2 contains 451 manually curated questions covering five core memory abilities for web agents.<n>We propose a suite of two memory methods: AgentRunbook-R, an efficient RAG-based memory with knowledge pools for raw state observations, events, and strategy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox.
  arXiv  Detail & Related papers  (2026-05-12T17:59:34Z)
• MEME: Multi-entity & Evolving Memory Evaluation [76.57263966646404]
  MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes.<n> evaluating six memory systems spanning three memory paradigms on 100 controlled episodes.<n>We find that all systems collapse on dependency reasoning under the default configuration.
  arXiv  Detail & Related papers  (2026-05-12T17:55:10Z)
• A Parametric Memory Head for Continual Generative Retrieval [52.66674234249913]
  Generative information retrieval (GenIR) consolidates retrieval into a single neural model that decodes document identifiers (docids) directly from queries.<n>We show that sequential adaptation improves retrieval on newly added documents but substantially degrades performance on earlier slices.<n>We propose post-adaptation memory tuning (PAMT), a memory-only stabilization stage that augments an adapted model with a modular parametric memory head.
  arXiv  Detail & Related papers  (2026-04-25T17:38:51Z)
• Spatial Metaphors for LLM Memory: A Critical Analysis of the MemPalace Architecture [0.0]
  MemPalace is an open-source AI memory system that applies the ancient method of loci (memory palace) spatial metaphor.<n>It claims state-of-the-art retrieval performance on the LongMemEval benchmark.<n>We find that MemPalace's headline retrieval performance is attributable primarily to its verbatim storage philosophy.
  arXiv  Detail & Related papers  (2026-04-23T04:59:16Z)
• D-Mem: A Dual-Process Memory System for LLM Agents [3.5426740232689604]
  We introduce D-Mem, a dual-process memory system.<n>It retains lightweight vector retrieval for routine queries while establishing an exhaustive Full Deliberation module as a high-fidelity fallback.<n>Experiments on the LoCoMo and RealTalk benchmarks using GPT-4o-mini and Qwen3-235B-Instruct demonstrate the efficacy of our approach.
  arXiv  Detail & Related papers  (2026-03-19T08:55:22Z)
• MemX: A Local-First Long-Term Memory System for AI Assistants [0.0]
  We present MemX, a local-first long-term memory system for AI with stability-oriented retrieval design.<n>MemX is implemented in Rust on top of lib and an OpenAI-compatible embedding API, providing persistent, searchable, and explainable memory for conversational agents.
  arXiv  Detail & Related papers  (2026-03-17T06:39:54Z)
• AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment [54.72137309071243]
  AlpsBench comprises 2,500 long-term interaction sequences curated from WildChat, paired with human-verified structured memories.<n>We define four pivotal tasks - personalized information extraction, updating, retrieval, and utilization - and establish protocols to evaluate the entire lifecycle of memory management.<n>Our benchmarking of frontier LLMs and memory-centric systems reveals that: (i) models struggle to reliably extract latent user traits; (ii) memory updating faces a performance ceiling even in the strongest models; (iii) retrieval accuracy declines sharply in the presence of large distractor pools; and (iv) while explicit memory mechanisms improve recall, they do not
  arXiv  Detail & Related papers  (2026-03-09T11:06:19Z)
• Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents [0.0]
  Persistent AI systems face a choice between passing full conversation histories to a long-context large language model (LLM) and maintaining a dedicated memory system that extracts and retrieves structured facts.<n>We compare a fact-based memory system built on the Mem0 framework against long-context LLM inference on three memory-centric benchmarks.
  arXiv  Detail & Related papers  (2026-03-05T05:01:30Z)
• Memory-Augmented Log Analysis with Phi-4-mini: Enhancing Threat Detection in Structured Security Logs [0.0]
  We propose textbfDM-RAG, a dual-memory retrieval-augmented generation framework for structured log analysis.<n>It integrates a short-term memory buffer for recent summaries and a long-term FAISS-indexed memory for historical patterns.<n>On the UNSW-NB15 dataset, DM-RAG 53.64% accuracy and 98.70% recall, surpassing fine-tuned and RAG baselines in recall.
  arXiv  Detail & Related papers  (2025-10-01T05:23:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.