Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings

• URL: http://arxiv.org/abs/2607.21962v1
• Date: Fri, 24 Jul 2026 04:19:45 GMT
• Title: Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings
• Abstract summary: A seeded life-script sampler emits facts with validity intervals volatility classes, and source channels before any text exists.<n>A verifier confirms every planted fact and questions are instantiated mechanically from the script.<n>The synthetic, fictionalized corpus embeds features absent from the benchmarks we survey.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Benchmarks for LLM-agent memory typically generate conversations first and extract answer keys afterwards -- with documented label-error and contamination problems -- and they overwhelmingly measure short interaction histories. We invert the pipeline: a seeded life-script sampler emits facts with validity intervals, volatility classes, and source channels before any text exists; an LLM renderer writes chat and email from per-event fact manifests; a fidelity verifier confirms every planted fact; and questions are instantiated mechanically from the script, so gold answers are script-valid by construction and separately validated for answerability. The synthetic, fictionalized corpus (~380 questions, 15 types) embeds features absent from the benchmarks we survey: per-fact validity intervals, sent/received trust distinctions, injection probes in a benign harness, and as-of-date question sets. Benchmarking five memory architectures against a no-memory control (fixed answerer, versioned LLM judge, three replicates, two horizons), we find backend rankings invert with history length: the budgeted curated-map memory that leads at three weeks loses recall of evicted content by nine weeks (96% to 72%) while a provenance-typed graph rises to 90%; the inversion is positive for all six users under complete cross-family re-judging (exact p=0.031). A full-rendered-history baseline ties or exceeds the best memory system at the short horizon but shows no judge-independent advantage at nine weeks, at about twice the read cost. Write-stage quality strongly correlates with downstream quality (weakly-written facts fail 24% vs 2%), and injection resistance tracked whether provenance boundaries survive representation. A layered architecture performs best among the memory systems in both regimes (96.8% short-horizon) and is released as Veracium, an open-source library, with the corpus generator and harness.

Related papers

• EM^2Mem: Event-Centric Multimodal Memory for Large Language Models [52.3165591032625]
  We propose EM2Mem, an event-centric multimodal memory framework that aligns heterogeneous evidence to event anchors during memory construction.<n>Across three long-video QA benchmarks, EM2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points.
  arXiv  Detail & Related papers  (2026-09-01T01:38:41Z)
• Agent Zero Memory: Provenance-Aware Long-Term Memory for LLM Agents [3.6666917625778193]
  Agent Zero Memory is a provenance-aware long-term memory system.<n>It distils a user's conversations, files, and connected sources into three parallel memory systems.
  arXiv  Detail & Related papers  (2026-08-30T06:55:59Z)
• Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration [0.05097809301149341]
  We introduce two full-text scientific-memory benchmarks, Public AI Memory (PAIM) and Public Transformers (PTr)<n>We evaluate eight memory/retrieval systems, including our own proposed Theoria, plus a no-retrieval baseline.<n>Our results show that memory leaderboards are not interpretable without the full protocol.
  arXiv  Detail & Related papers  (2026-07-18T15:09:58Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• Memory-Managed Long-Context Attention: A Preliminary Study of Editable Request-Local Memory [0.5371337604556311]
  We study memory-managed long-context attention, a research route that separates a fast recurrent or sparse backbone from explicit editable request-local memory slots.<n>Across structured synthetic tasks, token/chunk/sequence bridges, generated natural language, and local frozen-model diagnostics, pure fixed-state or pure sparse methods fail some overwrite, version, anti-pollution, or no-write-signal cases.
  arXiv  Detail & Related papers  (2026-06-27T11:38:43Z)
• ConvMemory v3: A Validity Context Layer for Conversational Memory via Target-Conditioned Relation Verification [0.0]
  ConvMemory v3 adds a validity context layer that detects and surfaces this update evidence.<n>The core mechanism is a dual-evidence gate that conditions a relation judgment on the specific target proposition.<n>On a synthetic multi-hop validity benchmark the gate reaches 90.12% +/- 1.73 accuracy.
  arXiv  Detail & Related papers  (2026-06-25T08:36:43Z)
• Echo-Memory: A Controlled Study of Memory in Action World Models [58.77172260403133]
  We present a controlled study of memory mechanisms in action-conditioned world models.<n>Existing memory designs are hard to compare because gains are entangled with backbone, training, retrieval, and evaluation differences.<n>We use Echo-Memory to fix the action-to-video interface and varies only how history is stored and read by the generator.
  arXiv  Detail & Related papers  (2026-06-08T17:54:10Z)
• Less Context, More Accuracy: A Bi-Temporal Memory Engine for LLM Agents Where a Lean Retrieved Context Beats the Full History [2.5401434059780468]
  Engram is a dual-process memory engine on a bi-temporal data model.<n>A fast write path appends episodes with no LLM on the critical path.<n>A hybrid read path fuses dense, lexical, graph, and recency/salience signals.
  arXiv  Detail & Related papers  (2026-06-05T11:43:56Z)
• Eywa: Provenance-Grounded Long-Term Memory for AI Agents [0.0]
  Existing memory systems collapse source evidence, extracted facts, retrieved context, and answer policy into one opaque prompt path.<n>We present Eywa, a provenance-grounded memory architecture built around evidence before belief.
  arXiv  Detail & Related papers  (2026-05-29T02:56:35Z)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction [72.1620416874118]
  Multimodal large language models are increasingly deployed as long-horizon agents.<n>Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions.<n>We formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle.
  arXiv  Detail & Related papers  (2026-05-28T04:27:20Z)
• Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory [88.81430082035617]
  LLM agents require a memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history.<n>We propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity.
  arXiv  Detail & Related papers  (2026-05-19T15:05:06Z)
• EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents [52.567469286881426]
  We introduce EMemBench, a programmatic benchmark for evaluating long-term memory of agents through interactive games.<n>Rather than using a fixed set of questions, EMemBench generates questions from each agent's own trajectory.<n>Each template computes verifiable ground truth from underlying game signals.
  arXiv  Detail & Related papers  (2026-01-23T12:09:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.