The Retriever Should Remember: Experience-Amortized Reranking for Long-Term Agent Memory

• URL: http://arxiv.org/abs/2608.22767v1
• Date: Mon, 24 Aug 2026 03:42:03 GMT
• Title: The Retriever Should Remember: Experience-Amortized Reranking for Long-Term Agent Memory
• Abstract summary: We introduce EARM, an experience-amortized reranking framework that treats previously acquired relevance scores as reusable retrieval experience.<n>EARM stores sparse query--memory relevance scores in an online matrix, learns their shared structure through causal matrix completion, and combines a small set of newly observed scores with estimated scores to rerank the remaining candidates.<n>Experiments on long-term conversational memory show that mixed observed-and-estimated reranking improves answer accuracy over semantic retrieval by up to 6.62%.
• Score: 29.20695058395373
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-term language-model agents accumulate memories across interactions, but their retrievers typically do not accumulate retrieval experience. Semantic retrieval is efficient, but embedding similarity does not always reflect whether a memory contains evidence relevant to the current query. Large language model (LLM) rerankers provide stronger query-conditioned relevance scores, yet stateless reranking repeatedly scores a large candidate pool and discards these scores after each query. We introduce EARM, an experience-amortized reranking framework that treats previously acquired LLM relevance scores as reusable retrieval experience. EARM stores sparse query--memory relevance scores in an online matrix, learns their shared structure through causal matrix completion, and combines a small set of newly observed scores with estimated scores to rerank the remaining candidates. The scoring budget decreases as experience accumulates, changing LLM reranking from a repeated per-query expense into a retrieval capability learned over an agent's lifetime. Experiments on long-term conversational memory show that mixed observed-and-estimated reranking improves answer accuracy over semantic retrieval by up to 6.62% and remains effective when only 17.5% of candidates receive direct LLM relevance scores, thereby substantially reducing the inference overhead of LLM reranking. These results motivate a broader view of agent memory: a long-lived agent should remember not only past content, but also how that content has proved useful for retrieval.

Related papers

• CABLE: Extending the Reach of Memory Retrieval via Complementary Antecedent-Based Linking and Expansion [21.692254412768936]
  Link construction should instead prioritize a sparse set of retriever-complementary associations.<n>We present CABLE, a plug-in augmentation that constructs links to extend the host retriever's direct semantic reach.<n>CABLE yields higher mean LLM-judge scores in every evaluated system-level setting.
  arXiv  Detail & Related papers  (2026-08-18T15:40:29Z)
• Exploratory and Assimilating Reflection: Reflective Recall Cycle for Long-term Memory [3.9297698032556596]
  LLM-based autonomous agents require external memory to overcome their statelessness and limited context window.<n>We propose Exploratory-Assimilating Reflection (EAR), a framework for high initial retrieval performance and sample-efficient adaptation.
  arXiv  Detail & Related papers  (2026-07-20T12:30:17Z)
• LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory [10.916121108457537]
  We introduce LazyMem, which deferring all memory construction to query time.<n>On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline.<n>It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline.
  arXiv  Detail & Related papers  (2026-07-17T12:49:57Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery [58.64329475683699]
  We argue that long-term memory should be evaluated as an auditable post-interaction artifact.<n>We instantiate this view in MEMPROBE, a benchmark in which a memory-equipped agent assists simulated users.<n>We find that successful assistance and recoverable memory behave as distinct capabilities.
  arXiv  Detail & Related papers  (2026-06-23T13:52:46Z)
• Thinking Ahead: Prospection-Guided Retrieval of Memory with Language Models [10.094292900770261]
  Long-horizon personalization requires dialogue assistants to retrieve user-specific facts from extended interaction histories.<n>In practice, many relevant facts often have low semanticsimilarity to the query under dense retrieval.<n>We introduce Prospection-Guided Retrieval (PGR), which decouples retrieval from how memories are stored.
  arXiv  Detail & Related papers  (2026-05-13T22:57:54Z)
• Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval [59.295767860331004]
  RF-Mem is a familiarity uncertainty-guided dual-path memory retriever.<n>It embeds human-like dual-process recognition into the retriever.<n>It consistently outperforms one-shot retrieval and full-context reasoning under fixed budget and latency constraints.
  arXiv  Detail & Related papers  (2026-03-10T06:31:44Z)
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory [89.65731902036669]
  Evo-Memory is a streaming benchmark and framework for evaluating self-evolving memory in large language model (LLM) agents.<n>We evaluate over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets.
  arXiv  Detail & Related papers  (2025-11-25T21:08:07Z)
• FIRST: Faster Improved Listwise Reranking with Single Token Decoding [56.727761901751194]
  First, we introduce FIRST, a novel listwise LLM reranking approach leveraging the output logits of the first generated identifier to directly obtain a ranked ordering of the candidates. Empirical results demonstrate that FIRST accelerates inference by 50% while maintaining a robust ranking performance with gains across the BEIR benchmark. Our results show that LLM rerankers can provide a stronger distillation signal compared to cross-encoders, yielding substantial improvements in retriever recall after relevance feedback.
  arXiv  Detail & Related papers  (2024-06-21T21:27:50Z)
• ReFIT: Relevance Feedback from a Reranker during Inference [109.33278799999582]
  Retrieve-and-rerank is a prevalent framework in neural information retrieval. We propose to leverage the reranker to improve recall by making it provide relevance feedback to the retriever at inference time.
  arXiv  Detail & Related papers  (2023-05-19T15:30:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.