Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks

• URL: http://arxiv.org/abs/2605.24060v1
• Date: Fri, 22 May 2026 02:53:07 GMT
• Title: Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks
• Abstract summary: Conversational-memory systems transform dialogue history into facts, summaries, timelines, and other source-linked descendants.<n>This raises an underspecified evaluation question: which stored form should receive retrieval credit?<n>We show that this scoring-target choice is often left implicit and can materially change benchmark conclusions.
• Score: 2.9089118242427627
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conversational-memory systems increasingly transform dialogue history into facts, summaries, timelines, and other source-linked descendants, so a single source turn can coexist with several derived memories in the same retrieval index. This raises an underspecified evaluation question: which stored form should receive retrieval credit? We show that this scoring-target choice is often left implicit and can materially change benchmark conclusions. We present TIAP, a fixed-output audit that rescores saved ranked outputs under three targets -- Raw, Source, and Canonical -- without rerunning retrieval. On LoCoMo and LongMemEval-S, switching only the credited target changes nDCG on 83.4--94.0 percent of shared queries, flips target orderings on Mem0 and MemoryOS transfer runs, and reverses parser-density recommendations. A 1,902-case semantic audit further shows that relaxed source-linked credit is fully justified only 29.2 percent of the time, despite high rubric reliability in a validation subset. These results reveal target noninvariance: conclusions about memory architectures can silently flip with a single benchmark-design choice. Conversational-memory papers should therefore define and report the scoring target explicitly.

Related papers

• InjecMEM: Memory Injection Attack on LLM Agent Memory Systems [31.104106193922956]
  InjecMEM is a novel memory injection attack paradigm.<n>We craft the injection with a retriever-agnostic anchor and an adversarial command.<n>We evaluate it across multiple memory systems and backbone models.
  arXiv  Detail & Related papers  (2026-08-24T16:37:50Z)
• Beyond Retrieval: Query-Conditioned Reuse of Long-Horizon Agent Trajectories [16.13133177152883]
  Retrieval can identify a past trajectory that may matter, yet it does not specify how an acting agent should use that trajectory after users, entities, constraints, or environment state have changed.<n>We identify this post-retrieval reuse step as a distinct bottleneck for long-horizon trajectory memory.<n>We instantiate the framework with query-conditioned reuse (QCR), a deliberately simple target-bound note that records a reusable procedure.
  arXiv  Detail & Related papers  (2026-08-13T05:39:49Z)
• TARL: Transaction-Aware Reliable Ledgers for Executable Memory Management in Long-Term Agents [20.123133746969398]
  We introduce TARL, a memory state update framework that maps each statement to one of five executable actions.<n> TARL identifies the affected memory, resolves its temporal scope, compares source reliability, and updates accepted, pending, and rejected ledgers.<n>We also introduce TARL-Mem, a benchmark with fine-grained action labels and next-state targets.
  arXiv  Detail & Related papers  (2026-08-04T14:02:55Z)
• Voice Memory for Agentic Speech Recognition [66.69623133635868]
  We present Voice Memory, a inference-only scheme for agentic speech recognition.<n>Asynchronously, a score-gated example revises that file through bounded edits.<n>Across ten HyPoradise domains with an open corrector, Voice Memory, weighted word error rate from 8.36% to 7.52%.
  arXiv  Detail & Related papers  (2026-07-29T02:42:56Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• ECHO: Prune to act, trace to learn with selective turn memory in agentic RL [9.429418772354397]
  ECHO is a selective turn-memory framework that addresses history collapse and traceable learning through source-indexed reconstruction.<n>On BrowseComp-Plus, ECHO reaches 43.4% held-out accuracy, outperforming GRPO (28.9%) and the rolling-summary baseline SUPO (36.1%), while using fewer turns and lower trajectory volume than SUPO.
  arXiv  Detail & Related papers  (2026-06-30T13:29:58Z)
• ConvMemory v3: A Validity Context Layer for Conversational Memory via Target-Conditioned Relation Verification [0.0]
  ConvMemory v3 adds a validity context layer that detects and surfaces this update evidence.<n>The core mechanism is a dual-evidence gate that conditions a relation judgment on the specific target proposition.<n>On a synthetic multi-hop validity benchmark the gate reaches 90.12% +/- 1.73 accuracy.
  arXiv  Detail & Related papers  (2026-06-25T08:36:43Z)
• MemMark: State-Evolution Attribution Watermarking for Agent Long-Term Memory Systems [13.133650395551507]
  Memory-backed agents need provenance that can survive leaked or migrated snapshots.<n>MemMark embeds an owner-controlled signal into latent memory-write decisions.<n>MemMark is feasible for long-term agent memory without surviving traces, trusted metadata, or utility-degrading.
  arXiv  Detail & Related papers  (2026-05-24T11:04:35Z)
• What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema [0.0]
  We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run.<n>We designed a small audit schema (five fields: benchmark identity, harness specification, inference settings, cost reporting, failure breakdown)<n>We score the disclosure of an agent run, not its correctness, and make no claim that disclosure implies a trustworthy result.
  arXiv  Detail & Related papers  (2026-05-20T17:02:36Z)
• Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory [88.81430082035617]
  LLM agents require a memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history.<n>We propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity.
  arXiv  Detail & Related papers  (2026-05-19T15:05:06Z)
• Goal-Oriented Reasoning for RAG-based Memory in Conversational Agentic LLM Systems [20.461904943047468]
  Goal-Mem is a goal-oriented reasoning framework for RAG-based agentic memory.<n>We show that Goal-Mem consistently improves performance on tasks requiring multi-hop reasoning and implicit inference.
  arXiv  Detail & Related papers  (2026-05-12T14:51:02Z)
• Belief Memory: Agent Memory Under Partial Observability [56.41506249481312]
  We propose BeliefMem, which shifts the memory paradigm from committing to a single conclusion per observation to retaining multiple candidate conclusions with their probabilities.<n> BeliefMem preserves the uncertainty that the deterministic paradigm discards, enabling the agent to act with high confidence.<n> Empirical evaluations on LoCoMo and ALFWorld benchmarks show that, even with limited data, BeliefMem achieves the best average performance.
  arXiv  Detail & Related papers  (2026-05-07T02:03:13Z)
• Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval [59.295767860331004]
  RF-Mem is a familiarity uncertainty-guided dual-path memory retriever.<n>It embeds human-like dual-process recognition into the retriever.<n>It consistently outperforms one-shot retrieval and full-context reasoning under fixed budget and latency constraints.
  arXiv  Detail & Related papers  (2026-03-10T06:31:44Z)
• MemGuide: Intent-Driven Memory Selection for Goal-Oriented Multi-Session LLM Agents [46.21840714172862]
  We introduce MemGuide, a two-stage framework for intent-driven memory selection.<n>MemGuide matches current dialogue context with stored intent descriptions in the memory bank.<n>Missing-Aligned Guided Filtering employs a chain-of-thought slot reasoner to enumerate unfilled slots, then uses a fine-tuned LLaMA-8B filter to re-rank the retrieved units.
  arXiv  Detail & Related papers  (2025-05-26T17:10:43Z)
• Unlocking the Power of SAM 2 for Few-Shot Segmentation [54.562050590453225]
  Few-Shot (FSS) aims to learn class-agnostic segmentation on few classes to segment arbitrary classes, but at the risk of overfitting.<n>Recently, SAM 2 has extended SAM by supporting video segmentation, whose class-agnostic matching ability is useful to FSS.<n>We design Pseudo Prompt Generator to encode pseudo query memory, matching with query features in a compatible way.<n>We further design Iterative Memory Refinement to fuse more query FG features into the memory, and devise a Support-Calibrated Memory Attention to suppress the unexpected query BG features in memory.
  arXiv  Detail & Related papers  (2025-05-20T09:02:53Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.