Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration

• URL: http://arxiv.org/abs/2607.16848v1
• Date: Sat, 18 Jul 2026 15:09:58 GMT
• Title: Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration
• Abstract summary: We introduce two full-text scientific-memory benchmarks, Public AI Memory (PAIM) and Public Transformers (PTr)<n>We evaluate eight memory/retrieval systems, including our own proposed Theoria, plus a no-retrieval baseline.<n>Our results show that memory leaderboards are not interpretable without the full protocol.
• Score: 0.05097809301149341
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-term memory is becoming a core component of LLM agents, but most memory benchmarks evaluate conversations or compact summaries, while research agents need to restore evidence from full scientific papers. We introduce two full-text scientific-memory benchmarks, Public AI Memory (PAIM; 81 papers, 66 questions) and Public Transformers (PTr; 252 papers, 98 questions). We evaluate eight memory/retrieval systems, including our own proposed Theoria, plus a no-retrieval baseline. Our results show that memory leaderboards are not interpretable without the full protocol: ingestion granularity, raw-text preservation, retrieval budget, retrieval modality, rubric audit, and judge choice all affect the outcome. For example, on PAIM Graphiti wins convincingly but uses 2.6M characters of retrieved context per query, and after controlling for retrieval budget the lead disappears. On PTr, for the systems where BM25 retrieval can be added cleanly, the sparse-dense hybrid is the single most significant intervention: hybrid variants of Simple RAG, Mem0, and Theoria tie for the lead within 0.03 points. Multi-judge and human side-by-side calibration show that LLM-as-a-judge rankings are consistent across frontier judges and agree with human evaluation, with an effective resolution of roughly one point on a ten-point scale. We argue that scientific memory should be evaluated as budgeted, modality-aware context restoration rather than as an unconstrained architecture leaderboard, and we release the datasets, harness, raw outputs, judgments, and scripts to reproduce our results and serve as tools for such evaluation. Our code is available at http://gitlab.com/quantellence/research/scientific-recall-bench , and the datasets are available at http://huggingface.co/datasets/quantellence/srb-data .

Related papers

• Continual Graph Memory for Adaptive Recommendation under Intent Drift [7.140905344515738]
  This paper studies adaptive recommendation under intent drift, where feedback from each recommendation outcome can reveal whether the relational evidence used for ranking is useful, missing, or misleading.<n>This paper proposes CGM-Rec, a continual graph memory framework for adaptive recommendation.
  arXiv  Detail & Related papers  (2026-09-04T02:32:20Z)
• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents [97.19033877543522]
  We present a controlled harness evaluation of memory substrates for memory-augmented agents.<n>We instrument 26 performance and efficiency metrics under a unified harness.<n>These findings motivate substrate routing as a necessary component of adaptive agent memory systems.
  arXiv  Detail & Related papers  (2026-08-15T03:29:48Z)
• Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings [0.0]
  A seeded life-script sampler emits facts with validity intervals volatility classes, and source channels before any text exists.<n>A verifier confirms every planted fact and questions are instantiated mechanically from the script.<n>The synthetic, fictionalized corpus embeds features absent from the benchmarks we survey.
  arXiv  Detail & Related papers  (2026-07-24T04:19:45Z)
• MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery [58.64329475683699]
  We argue that long-term memory should be evaluated as an auditable post-interaction artifact.<n>We instantiate this view in MEMPROBE, a benchmark in which a memory-equipped agent assists simulated users.<n>We find that successful assistance and recoverable memory behave as distinct capabilities.
  arXiv  Detail & Related papers  (2026-06-23T13:52:46Z)
• eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents [1.8047694351309203]
  We present eMEM, a hybrid graph-based memory system for embodied agents operating in physical environments.<n>eMEM fills this gap with a multi-index architecture (ITE for structured storage, hlib for approximate nearest neighbour semantic search, and an R-tree for spatial queries)<n>We also introduce eMEM-Bench v1, a benchmark we construct over ProcTHOR-10K scenes for embodied memory evaluation.
  arXiv  Detail & Related papers  (2026-06-02T09:22:12Z)
• Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory [50.857546269660276]
  We introduce SeqMem-Eval, a diagnostic evaluation framework for sequentially evolving large language models (LLMs) memory.<n>Rather than focusing only on final performance, SeqMem-Eval evaluates how memory states evolve, generalize, consolidate experience, and retain useful information during sequential inference.
  arXiv  Detail & Related papers  (2026-05-14T20:15:22Z)
• From Unstructured Recall to Schema-Grounded Memory: Reliable AI Memory via Iterative, Schema-Aware Extraction [39.146761527401424]
  Persistent AI memory is often reduced to a retrieval problem.<n>This paper argues that reliable external AI memory must be schema-grounded harnesses.<n>We present an iterative, schema-aware write path that decomposes memory ingestion into object detection, field detection, and field-value extraction.
  arXiv  Detail & Related papers  (2026-04-30T14:14:02Z)
• When to Forget: A Memory Governance Primitive [0.0]
  Memory Worth is a two-counter per-memory signal that tracks how often a memory co-occurs with successful versus failed outcomes.<n>We prove that MW converges almost surely to the conditional success probability p+(m) = Pr[y_t = +1 | m in M_t] under a stationary retrieval regime.<n>We argue this is still a useful operational signal for memory governance, and we validate it empirically in a controlled synthetic environment.
  arXiv  Detail & Related papers  (2026-04-13T19:54:14Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory [6.493102878734374]
  We analyze how performance differences manifest across write strategies, retrieval methods, and memory utilization behavior.<n>On LoCoMo, retrieval method is the dominant factor.<n>Failure analysis shows that performance breakdowns most often manifest at the retrieval stage rather than at utilization.
  arXiv  Detail & Related papers  (2026-03-02T23:47:23Z)
• NextMem: Towards Latent Factual Memory for LLM-based Agents [58.35585202907478]
  NextMem is a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory.<n>Extensive experiments demonstrate that NextMem achieves superior performance.
  arXiv  Detail & Related papers  (2026-02-26T14:35:27Z)
• Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions [55.19217798774033]
  Memory is a fundamental component of AI systems, underpinning large language models (LLMs)-based agents.<n>In this survey, we first categorize memory representations into parametric and contextual forms.<n>We then introduce six fundamental memory operations: Consolidation, Updating, Indexing, Forgetting, Retrieval, and Compression.
  arXiv  Detail & Related papers  (2025-05-01T17:31:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.