Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One

• URL: http://arxiv.org/abs/2606.25449v2
• Date: Mon, 29 Jun 2026 07:03:19 GMT
• Title: Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One
• Abstract summary: A language model's memory can be worse than no memory at all.<n>A memory that keeps a wrong conclusion but drops the work behind it makes the model emit the stale value as a confident answer.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A language model's memory can be worse than no memory at all. A memory that keeps a wrong conclusion but drops the work behind it makes the model emit the stale value as a confident answer, where an empty memory would make it abstain; we call this brittle memory. We measure it with reclaim evaluation: compress a drifted interaction at a fixed budget, then test whether a correction recovers the known answer, scored against ground truth with no judge. Correctability is bottlenecked not by capability but by whether the answer-determining source survives compression, so an 8B model and a frontier one wall in the same place. Across eight models a lossy memory is never better than an empty one, and strictly worse on those disposed to answer rather than abstain. A one-line source-first policy, keep the recomputable source and drop the re-derivable conclusion, restores correctability at equal budget where the answer-determining source is compact and identifiable; a length-matched control rules out added text, and a deployable one-prompt form reclaims 0.49-0.88, rising toward the oracle's 1.00 when a frontier model writes the note. The failure compounds through a memory loop and replicates on three deployed memory systems and on real dialogue (MultiWOZ), with a located boundary past which the fix fails silently unless the note records its completeness. This is a controlled study of a mechanism: judge-free exact scoring, matched-budget controls, and validators built to come out false; we release the harness, the paired memory conditions, and these validators.

Related papers

• The Cost of Compression: A Rate-Distortion Limit on Factual Hallucination [5.158280959759542]
  Factual hallucination in closed-book question answering is often treated as a coverage problem.<n>We study this effect through a simple coverage--compression model of factual recall.<n>We study the predicted signatures with theory-implied simulations and controlled fact-injection probes in modern language models.
  arXiv  Detail & Related papers  (2026-09-10T18:37:44Z)
• Towards a Formal Definition of Agent Memory: Basis, Span, Optimality, and the Sequential Memory Problem [12.078734827417685]
  There is no unified formal account of what a memory is or when it is optimal.<n>The central idea is that memory is a basis, knowledge is its span, and answerability is a coverage problem.<n>The optimal memory is then the capacity-constrained maximizer of expected coverage.
  arXiv  Detail & Related papers  (2026-08-12T04:54:26Z)
• From Faulty Memories to Corrected Actions: Dependency-Guided Rollback Repair for Memory-Augmented Agents [19.125716338799545]
  textbfdependency-guided rollback repair builds a typed memory-to-action graph from provenance, traces explicit downstream dependencies, and selectively replays only answer-relevant affected computation.<n>We evaluate this approach on a 150-case controlled benchmark spanning three tool-use domains and four memory failure types.
  arXiv  Detail & Related papers  (2026-08-11T05:19:55Z)
• Voice Memory for Agentic Speech Recognition [66.69623133635868]
  We present Voice Memory, a inference-only scheme for agentic speech recognition.<n>Asynchronously, a score-gated example revises that file through bounded edits.<n>Across ten HyPoradise domains with an open corrector, Voice Memory, weighted word error rate from 8.36% to 7.52%.
  arXiv  Detail & Related papers  (2026-07-29T02:42:56Z)
• Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory [32.094723684433895]
  Long-term memory systems store what a user says in an external store and retrieve it when a related query arrives.<n>This interface rests on an assumption so natural that it is rarely stated: a memory that is needed will resemble the query that needs it.<n>We call this failure mode the implicit-association blind spot and introduce InMind, a benchmark spanning ten life domains.
  arXiv  Detail & Related papers  (2026-07-27T12:42:12Z)
• Forget to Improve: On-Device LLM-Agent Continual Learning via Budget-Curated Memory [7.254575731198877]
  On-device language-model agents improve by accumulating experience in retrieved memory rather than by updating weights.<n>We propose sys, a single net-value-per-byte score that governs an agent's experience-memory lifecycle.
  arXiv  Detail & Related papers  (2026-06-23T19:42:07Z)
• Learning What to Remember: A Cognitively Grounded Multi-Factor Value Model for Agentic Memory [1.2127875744950842]
  Long-running LLM agents accumulate interaction histories far larger than any context window.<n>Production systems answer with semantic similarity or recency.<n>We propose a multi-factor memory function V(m)=sum_i w_i f_i(m) over seven interpretable factors.
  arXiv  Detail & Related papers  (2026-06-11T06:17:31Z)
• Echo-Memory: A Controlled Study of Memory in Action World Models [58.77172260403133]
  We present a controlled study of memory mechanisms in action-conditioned world models.<n>Existing memory designs are hard to compare because gains are entangled with backbone, training, retrieval, and evaluation differences.<n>We use Echo-Memory to fix the action-to-video interface and varies only how history is stored and read by the generator.
  arXiv  Detail & Related papers  (2026-06-08T17:54:10Z)
• Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle [0.0]
  Reference-free faithfulness metrics verify each atomic claim a model makes against ground truth.<n>We show they share a blind spot: they measure only precision, are the stated claims supported?<n>We make this measurable using Formula 1 telemetry, a domain where strategic ground truth is derived deterministically.
  arXiv  Detail & Related papers  (2026-06-08T11:56:25Z)
• TOKI: A Bitemporal Operator Algebra for Contradiction Resolution in LLM-Agent Persistent Memory [22.85129722207174]
  We show that contradiction resolution is write-time control and make the missing contract explicit.<n> TOKI types the fourjudicats as one family of bitemporal operators over a dual-row schema.<n>A verdict matrix over eight systems localizes the gap: every baseline that keeps a language-model judge on the write path admits at least one of three write-time anomalies.
  arXiv  Detail & Related papers  (2026-06-04T14:46:52Z)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction [72.1620416874118]
  Multimodal large language models are increasingly deployed as long-horizon agents.<n>Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions.<n>We formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle.
  arXiv  Detail & Related papers  (2026-05-28T04:27:20Z)
• MemGuard: Preventing Memory Contamination in Long-Term Memory-Augmented Large Language Models [56.31411457917676]
  We introduce MemGuard, a type-aware memory framework that preserves functional memory boundaries during memory construction and retrieval.<n>Across hallucination and long-horizon conversation benchmarks, MemGuard improves memory reliability by up to 28.27% while retrieving up to 5.8x fewer memory tokens than prior methods.
  arXiv  Detail & Related papers  (2026-05-27T06:04:19Z)
• Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory [120.68750043918249]
  We show that memory should preserve the distinctions that matter for decisions, not descriptions.<n>We propose DeMem, an online memory learner that refines its partition only when data certify that a shared state would induce decision conflict.
  arXiv  Detail & Related papers  (2026-05-11T17:20:58Z)
• When Stored Evidence Stops Being Usable: Scale-Conditioned Evaluation of Agent Memory [4.425240080125479]
  We present a scale-conditioned evaluation protocol for agent memory under evidence-preserving growth.<n>For each query, task evidence is held fixed while irrelevant sessions are added.<n>The protocol logs agent--memory trajectories and reports four diagnostics.
  arXiv  Detail & Related papers  (2026-05-08T06:22:31Z)
• Belief Memory: Agent Memory Under Partial Observability [56.41506249481312]
  We propose BeliefMem, which shifts the memory paradigm from committing to a single conclusion per observation to retaining multiple candidate conclusions with their probabilities.<n> BeliefMem preserves the uncertainty that the deterministic paradigm discards, enabling the agent to act with high confidence.<n> Empirical evaluations on LoCoMo and ALFWorld benchmarks show that, even with limited data, BeliefMem achieves the best average performance.
  arXiv  Detail & Related papers  (2026-05-07T02:03:13Z)
• Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval [59.295767860331004]
  RF-Mem is a familiarity uncertainty-guided dual-path memory retriever.<n>It embeds human-like dual-process recognition into the retriever.<n>It consistently outperforms one-shot retrieval and full-context reasoning under fixed budget and latency constraints.
  arXiv  Detail & Related papers  (2026-03-10T06:31:44Z)
• Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution [52.76038908826961]
  We propose $textbfReMe$ ($textitRemember Me, Refine Me$) to bridge the gap between static storage and dynamic reasoning.<n>ReMe innovates across the memory lifecycle via three mechanisms: $textitmulti-faceted distillation$, which extracts fine-grained experiences.<n>Experiments on BFCL-V3 and AppWorld demonstrate that ReMe establishes a new state-of-the-art in agent memory system.
  arXiv  Detail & Related papers  (2025-12-11T14:40:01Z)
• Agentic Learner with Grow-and-Refine Multimodal Semantic Memory [50.81667005063605]
  ViLoMem is a dual-stream memory framework that constructs compact, schema-based memory.<n>It encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences.
  arXiv  Detail & Related papers  (2025-11-26T18:55:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.