What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory

• URL: http://arxiv.org/abs/2609.08279v1
• Date: Tue, 08 Sep 2026 05:48:10 GMT
• Title: What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory
• Abstract summary: We introduce the restore counterfactual, a per-question paired intervention that reinstates the question's gold evidence in the read-time context.<n>We evaluate FIFO, random, redundancy-aware, and LLM-importance eviction on LongMemEval-S at three budgets and under two retrieval regimes.
• Score: 1.6558929815405155
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the resulting loss in accuracy, but do not distinguish irreversible losses caused by eviction from recoverable retrieval failures. We introduce the restore counterfactual, a per-question paired intervention that reinstates the question's gold evidence in the read-time context and reruns the same reader. Combining the change in correctness with whether the evidence was retained after eviction classifies each oracle-answerable error as recoverable, irreversible, or residual; in the residual case, the answer remains incorrect after restoration. We evaluate FIFO, random, redundancy-aware, and LLM-importance eviction on LongMemEval-S at three budgets and under two retrieval regimes, using GPT-4o-mini as the primary reader and judge and GPT-5.4-mini as a robustness reader. Under top-k retrieval at an 80k-token budget, the irreversible share among errors corrected by restoration is 0.67-0.73 for FIFO, random, and redundancy-aware eviction, compared with 0.60 for LLM-importance. At 8k tokens, it reaches 1.00 for all four policies. Recoverable errors occur under top-k retrieval at 80k tokens but are absent under forced-gold injection by construction, so budget-accuracy results are not directly comparable unless the retrieval regime is reported. An exploratory matched-accuracy analysis detects no difference in irreversible rate among accuracy-matched policy pairs at a resolution of 1.2-6 percentage points. The same analysis detects the deliberately destructive control. To our knowledge, this is the first per-item, per-question restore-counterfactual audit of eviction for external agent-memory stores on a standard conversational benchmark.

Related papers

• When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory [0.0]
  Provenance links keep the evidence behind an inherited belief reachable; an agent with a verification budget must still choose which links to inspect.<n>We study a consolidated memory that states a decision constraint and whose source record has since been superseded by a record that withdraws it.<n>16 language models rarely re-verified a constraint that read as settled.<n>Once the constraint had been superseded, produced stale-consistent decisions in 77.3%, 74.7% and 74.7% of episodes across a primary run, a replication and a held-out domain.
  arXiv  Detail & Related papers  (2026-08-26T09:04:21Z)
• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• Measure, Don't Optimize: Forecasting Recovery in LLM Unlearning [57.920153607557744]
  J-Access is an inference-time audit that measures how often target concepts remain accessible along the model's output pathway.<n>We audit 398 public unlearned models spanning eight unlearning methods.
  arXiv  Detail & Related papers  (2026-08-11T20:16:20Z)
• Independent Patch Verification for Coding Agents with a Bidirectional Reconstruct-and-Verify Framework [49.16055123488827]
  coding agents powered by large language models can now generate code patches directly from bug reports.<n>No mechanism independently verifies whether it truly resolves the reported problem.<n>We propose RETRACE, a training-free post-generation verification framework.
  arXiv  Detail & Related papers  (2026-08-09T22:59:13Z)
• What Would Fix This RAG Failure? Auditing Counterfactual Response with Paired Evidence Interventions [0.0]
  We introduce Pair-ID, an offline audit that holds one query, retrieval state, and reader constant, then crosses two operations, adding missing support and deleting verified nonsupport, to measure a same-failure counterfactual response vector.<n>A complete funnel over 19,981 benchmark queries identifies 11,105 eligible Qwen failures, from which a prospectively fixed SHA-256 ordering selects 1,200 before generating any sampled response.<n>Results show that evidence sensitivity occurs at meaningful rates in the hash-selected eligible-failure sample, is only partially predictable from the observed failure, and is conditional on the reader.
  arXiv  Detail & Related papers  (2026-08-09T22:43:27Z)
• When Do Corrective Features Help? An Agent for Corrective Feature Discovery on Black-Box Forecasters [44.11700214719062]
  We study corrective feature discovery: mining interpretable features of a frozen forecaster's residual to drive a lightweight post-hoc corrector.<n>We present CRAFTER, which keeps the backbone frozen and mines its residual with two complementary generators.<n>A single validation-grounded gate accepts or rejects every candidate regardless of its origin, and a validation-selected corrector applies the accepted features or leaves the forecast unchanged.
  arXiv  Detail & Related papers  (2026-08-05T07:48:35Z)
• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• Error Certificates for KV-Cache Eviction via Randomized Design [3.860898721527665]
  KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest.<n>We prove that this design cannot know what it destroyed.<n>We show that Randomized eviction restores identifiability.
  arXiv  Detail & Related papers  (2026-07-23T16:16:59Z)
• Mean Time to Remediate Is Not a Fielding Model: A Cadence Audit for Enterprise Vulnerability Management [51.56484100374058]
  This paper introduces a remediation-cadence audit for enterprise vulnerability management.<n>It records routine mean lag, release period, release fraction, cohort geometry, emergency/routine split, non-fielding delay, local residual-pressure evidence, and declared rate scenario.
  arXiv  Detail & Related papers  (2026-07-05T21:23:26Z)
• Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning [15.231928045776293]
  sevra is a serving-layer controller that decides whether to preserve a frozen solver's initial answer or invoke active verification.<n>On mathfive, selective verification reaches 76.3% accuracy, compared with 75.5% for always verifying.<n>On CommonsenseQA, always-on verification hurts, while Self-Consistency@5 improves accuracy about five times the realized token cost.
  arXiv  Detail & Related papers  (2026-06-18T05:25:43Z)
• SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation [6.604874054866016]
  We present SURE-RAG, a transparent aggregation protocol built on the observation that evidence sufficiency is a set-level property.<n>A shared pair-level claim-evidence verifier produces local relation distributions, which SURE-RAG aggregates into interpretable answer-level signals.<n>We evaluate on HotpotQA-RAG v3, a controlled multi-hop benchmark, under an artifact-aware protocol.
  arXiv  Detail & Related papers  (2026-05-05T09:05:40Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.