Evaluating Memory Structure in LLM Agents

• URL: http://arxiv.org/abs/2602.11243v1
• Date: Wed, 11 Feb 2026 17:32:23 GMT
• Title: Evaluating Memory Structure in LLM Agents
• Authors: Alina Shutova, Alexandra Olenina, Ivan Vinogradov, Anton Sinitsin,
• Abstract summary: StructMemEval is a benchmark that tests the agent's ability to organize its long-term memory, not just factual recall.<n>We gather a suite of tasks that humans solve by organizing their knowledge in a specific structure.<n>Our experiments show that simple retrieval-augmented LLMs struggle with these tasks, whereas memory agents can reliably solve them if prompted how to organize their memory.
• Score: 39.55225799412317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern LLM-based agents and chat assistants rely on long-term memory frameworks to store reusable knowledge, recall user preferences, and augment reasoning. As researchers create more complex memory architectures, it becomes increasingly difficult to analyze their capabilities and guide future memory designs. Most long-term memory benchmarks focus on simple fact retention, multi-hop recall, and time-based changes. While undoubtedly important, these capabilities can often be achieved with simple retrieval-augmented LLMs and do not test complex memory hierarchies. To bridge this gap, we propose StructMemEval - a benchmark that tests the agent's ability to organize its long-term memory, not just factual recall. We gather a suite of tasks that humans solve by organizing their knowledge in a specific structure: transaction ledgers, to-do lists, trees and others. Our initial experiments show that simple retrieval-augmented LLMs struggle with these tasks, whereas memory agents can reliably solve them if prompted how to organize their memory. However, we also find that modern LLMs do not always recognize the memory structure when not prompted to do so. This highlights an important direction for future improvements in both LLM training and memory frameworks.

Related papers

• Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents [57.38404718635204]
  Large language model (LLM) agents face fundamental limitations in long-horizon reasoning due to finite context windows.<n>Existing methods typically handle long-term memory (LTM) and short-term memory (STM) as separate components.<n>We propose Agentic Memory (AgeMem), a unified framework that integrates LTM and STM management directly into the agent's policy.
  arXiv  Detail & Related papers  (2026-01-05T08:24:16Z)
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory [89.65731902036669]
  Evo-Memory is a streaming benchmark and framework for evaluating self-evolving memory in large language model (LLM) agents.<n>We evaluate over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets.
  arXiv  Detail & Related papers  (2025-11-25T21:08:07Z)
• Evaluating Long-Term Memory for Long-Context Question Answering [100.1267054069757]
  We present a systematic evaluation of memory-augmented methods using LoCoMo, a benchmark of synthetic long-context dialogues annotated for question-answering tasks.<n>Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-10-27T18:03:50Z)
• Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning [89.55738101744657]
  Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of NLP tasks, but they remain fundamentally stateless.<n>We present Memory-R1, a reinforcement learning framework that equips LLMs with the ability to actively manage and utilize external memory.
  arXiv  Detail & Related papers  (2025-08-27T12:26:55Z)
• On the Structural Memory of LLM Agents [20.529239764968654]
  Memory plays a pivotal role in enabling large language model(LLM)-based agents to engage in complex and long-term interactions.<n>This paper investigates how memory structures and memory retrieval methods affect the performance of LLM-based agents.
  arXiv  Detail & Related papers  (2024-12-17T04:30:00Z)
• Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks [42.22616978679253]
  We introduce Sequence Order Recall Tasks (SORT), which we adapt from tasks used to study episodic memory in cognitive psychology. SORT requires LLMs to recall the correct order of text segments, and provides a general framework that is both easily extendable and does not require any additional annotations. Based on a human experiment with 155 participants, we show that humans can recall sequence order based on long-term memory of a book.
  arXiv  Detail & Related papers  (2024-10-10T17:17:38Z)
• Needle in the Haystack for Memory Based Large Language Models [31.885539843977472]
  Current large language models (LLMs) often perform poorly on simple fact retrieval tasks. We investigate if coupling a dynamically adaptable external memory to a LLM can alleviate this problem. We demonstrate that the external memory of Larimar, which allows fast write and read of an episode of text samples, can be used at test time to handle contexts much longer than those seen during training.
  arXiv  Detail & Related papers  (2024-07-01T16:32:16Z)
• MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory [49.96019697955383]
  We introduce MemLLM, a novel method of enhancing large language models (LLMs) by integrating a structured and explicit read-and-write memory module.<n>Our experiments indicate that MemLLM enhances the LLM's performance and interpretability, in language modeling in general and knowledge-intensive tasks in particular.
  arXiv  Detail & Related papers  (2024-04-17T18:13:16Z)
• Empowering Working Memory for Large Language Model Agents [9.83467478231344]
  This paper explores the potential of applying cognitive psychology's working memory frameworks to large language models (LLMs) An innovative model is proposed incorporating a centralized Working Memory Hub and Episodic Buffer access to retain memories across episodes. This architecture aims to provide greater continuity for nuanced contextual reasoning during intricate tasks and collaborative scenarios.
  arXiv  Detail & Related papers  (2023-12-22T05:59:00Z)
• RET-LLM: Towards a General Read-Write Memory for Large Language Models [53.288356721954514]
  RET-LLM is a novel framework that equips large language models with a general write-read memory unit. Inspired by Davidsonian semantics theory, we extract and save knowledge in the form of triplets. Our framework exhibits robust performance in handling temporal-based question answering tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:53:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.