MemFail: Stress-Testing Failure Modes of LLM Memory Systems

• URL: http://arxiv.org/abs/2605.26667v1
• Date: Tue, 26 May 2026 08:03:55 GMT
• Title: MemFail: Stress-Testing Failure Modes of LLM Memory Systems
• Abstract summary: Large language model (LLM) agents increasingly rely on external memory systems to remain consistent across long-horizon interactions.<n>Existing benchmarks report aggregate question-answering accuracy and treat memory systems as black boxes.<n>We introduce MemFail, a diagnostic benchmark that isolates the failure modes of modern LLM memory systems.
• Score: 69.80981631587501
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) agents increasingly rely on external memory systems to remain consistent across long-horizon interactions, but little empirical work has been done to understand the specific failure modes and design choices that these systems present. Existing benchmarks report aggregate question-answering accuracy and treat memory systems as black boxes, making it impossible to attribute an incorrect answer to a particular failure mode of the system. We introduce MemFail, a diagnostic benchmark that isolates the failure modes of modern LLM memory systems. We begin by formalizing memory systems as the composition of three canonical operations -- summarization, storage, and retrieval -- and identify the potential failure modes induced by each. Based on these hypothesized failure modes, we construct five datasets spanning four tasks, each adversarially designed to test a specific operation of a memory system. Using these datasets, we evaluate four state-of-the-art memory systems on MemFail and demonstrate how MemFail can be used to empirically understand the tradeoffs induced by differences in memory system architectures.

Related papers

• MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use [30.64715962774191]
  Existing memory benchmarks mainly evaluate whether information is correctly extracted, stored, and retrieved.<n>We identify memory-induced cognitive traps: even faithfully recorded and semantically relevant memories can distort model reasoning.<n>We propose AdaptiveMem, a simple yet effective inference-time method that instructs LLMs to avoid memory traps.
  arXiv  Detail & Related papers  (2026-08-20T16:00:17Z)
• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents [97.19033877543522]
  We present a controlled harness evaluation of memory substrates for memory-augmented agents.<n>We instrument 26 performance and efficiency metrics under a unified harness.<n>These findings motivate substrate routing as a necessary component of adaptive agent memory systems.
  arXiv  Detail & Related papers  (2026-08-15T03:29:48Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems [24.22940060094778]
  We study the new problem of error tracing and attribution in LLM memory systems.<n>We propose a novel framework that transforms memory pipelines into executable memory evolution graphs.<n>We then construct MemTraceBench, a benchmark collected from memory systems such as Long-Context, RAG, Mem0, and EverMemOS.
  arXiv  Detail & Related papers  (2026-05-27T16:53:53Z)
• Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory [50.857546269660276]
  We introduce SeqMem-Eval, a diagnostic evaluation framework for sequentially evolving large language models (LLMs) memory.<n>Rather than focusing only on final performance, SeqMem-Eval evaluates how memory states evolve, generalize, consolidate experience, and retain useful information during sequential inference.
  arXiv  Detail & Related papers  (2026-05-14T20:15:22Z)
• MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents [78.95081012334116]
  Large Language Models (LLMs) with persistent memory enhances interaction continuity and personalization but introduces new safety risks.<n> contaminated or biased memory accumulation can trigger abnormal agent behaviors.<n>MemeEvoBench is the first benchmark evaluating memory safety in LLM agents against adversarial memory injection, noisy tool outputs, and biased feedback.
  arXiv  Detail & Related papers  (2026-04-17T07:29:52Z)
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions [22.190297901876278]
  We identify four core competencies essential for memory agents: accurate retrieval, test-time learning, long-range understanding, and selective forgetting.<n>Existing benchmarks either rely on limited context lengths or are tailored for static, long-context settings like book-based QA.<n>We introduce MemoryAgentBench, a new benchmark specifically designed for memory agents.
  arXiv  Detail & Related papers  (2025-07-07T17:59:54Z)
• MemOS: A Memory OS for AI System [116.87568350346537]
  Large Language Models (LLMs) have become an essential infrastructure for Artificial General Intelligence (AGI)<n>Existing models mainly rely on static parameters and short-lived contextual states, limiting their ability to track user preferences or update knowledge over extended periods.<n>MemOS is a memory operating system that treats memory as a manageable system resource.
  arXiv  Detail & Related papers  (2025-07-04T17:21:46Z)
• A Comprehensive Quantification of Inconsistencies in Memory Dumps [13.796554685139855]
  We develop a system to track all write operations performed by the OS kernel during a memory acquisition process.<n>We quantify how different acquisition modes, file systems, and hardware targets influence the frequency of kernel writes during the dump.
  arXiv  Detail & Related papers  (2025-03-19T10:02:54Z)
• On the Structural Memory of LLM Agents [20.529239764968654]
  Memory plays a pivotal role in enabling large language model(LLM)-based agents to engage in complex and long-term interactions.<n>This paper investigates how memory structures and memory retrieval methods affect the performance of LLM-based agents.
  arXiv  Detail & Related papers  (2024-12-17T04:30:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.