Beyond Similarity: Trustworthy Memory Search for Personal AI Agents

• URL: http://arxiv.org/abs/2606.06054v1
• Date: Thu, 04 Jun 2026 11:54:29 GMT
• Title: Beyond Similarity: Trustworthy Memory Search for Personal AI Agents
• Abstract summary: We study memory search as a trust boundary in personal AI agents.<n>MemGate is a lightweight and deployable memory plug-in for trustworthy memory search.
• Score: 25.265839311088516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Personal AI agents increasingly rely on long-term memory to provide persistent personalization across sessions. However, existing memory pipelines are largely driven by semantic similarity: memory data close to the current query is retrieved and injected into the model context. This creates a critical trustworthiness gap, since a semantically related memory may still be contextually inappropriate, leading to threats such as cross-domain leakage, sycophancy, tool-call drift, or memory-induced jailbreaks. In this paper, we study memory search as a trust boundary in personal AI agents. We evaluate representative agentic memory frameworks, including A-Mem, Mem0, and MemOS, together with OpenClaw, a real-world personal-agent environment with persistent state and tool-use capability. Our results show that long-term memory is not merely a utility layer, but a durable control channel that can reshape how agents interpret tasks and execute actions, leaving them highly susceptible to the aforementioned threats. To mitigate these vulnerabilities, we propose MemGate, a lightweight and deployable memory plug-in for trustworthy memory search, with only 9M parameters and a 35.1MB footprint. MemGate is inserted between the vector memory store and the backbone LLM, requiring no LLM modification, memory-database rewriting, or inference-time LLM judge. It applies a query-conditioned neural gate to candidate memory representations, turning raw similarity search into task-conditioned memory admission. Across multiple mainstream memory frameworks, real-world agent settings, and diverse LLM backbones, MemGate reduces memory-induced threats while preserving long-term memory utility.

Related papers

• InjecMEM: Memory Injection Attack on LLM Agent Memory Systems [31.104106193922956]
  InjecMEM is a novel memory injection attack paradigm.<n>We craft the injection with a retriever-agnostic anchor and an adversarial command.<n>We evaluate it across multiple memory systems and backbone models.
  arXiv  Detail & Related papers  (2026-08-24T16:37:50Z)
• Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents [72.3752981234916]
  Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks.<n>We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process.<n>MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5-20%.
  arXiv  Detail & Related papers  (2026-07-15T08:32:40Z)
• MemSyco-Bench: Benchmarking Sycophancy in Agent Memory [32.49512092540906]
  MemSyco-Bench is a benchmark for evaluating memory-induced sycophancy in agent systems.<n>MemSyco-Bench measures when memory should influence a decision and how valid memory should be used.
  arXiv  Detail & Related papers  (2026-07-01T15:30:33Z)
• MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery [58.64329475683699]
  We argue that long-term memory should be evaluated as an auditable post-interaction artifact.<n>We instantiate this view in MEMPROBE, a benchmark in which a memory-equipped agent assists simulated users.<n>We find that successful assistance and recoverable memory behave as distinct capabilities.
  arXiv  Detail & Related papers  (2026-06-23T13:52:46Z)
• Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents [51.30250860677378]
  We propose MRAgent, a framework that combines an associative memory graph with an active reconstruction mechanism.<n>Running on a Cue-Tag-Content graph, our active reconstruction mechanism integrates reasoning directly into memory access.<n>Experiments on the LoCoMo benchmark and LongMemEval benchmark demonstrate significant improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T11:29:46Z)
• SubtleMemory: A Benchmark for Fine-Grained Relational Memory Discrimination in Long-Horizon AI Agents [38.778004697710855]
  We introduce SubtleMemory, a benchmark for fine-grained relational memory discrimination in long-running AI agents.<n>We evaluate six standalone memory systems, two Claw-style agents with native memory modules, and three Claw-style agents with plugin memory modules.
  arXiv  Detail & Related papers  (2026-06-04T06:43:11Z)
• MemGym: a Long-Horizon Memory Environment for LLM Agents [69.79226770543049]
  We present MemGym, a benchmark for agentic memory that unifies existing agent gyms and in-house memory-grounded pipelines.<n>MemGym reports memory-isolated scores that decouple memory performance from reasoning, retrieval, and tool-use ability.<n>Our synthetic pipelines for MEMGYM-CODEQA and MEMGYM-DR are length-controllable, ablation-verified at every stage, and tightly aligned with downstream scenarios.
  arXiv  Detail & Related papers  (2026-05-20T07:25:33Z)
• Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents [18.83666486835137]
  In deployment, a single agent serves many independent tasks over a long horizon, and memory accumulated during earlier tasks can affect behavior on later, unrelated ones.<n>We introduce a trigger-probe protocol that evaluates a fixed probe set against read-only memory snapshots at varying prefix lengths.<n>We apply this protocol across three deployment scenarios spanning records, memos, forms, and email correspondence and eight memory architectures.
  arXiv  Detail & Related papers  (2026-05-18T04:06:34Z)
• Causal Intervention-Based Memory Selection for Long-Horizon LLM Agents [0.33842793760651557]
  We propose a causal memory-selection technique that estimates how candidate memories affect the model's answer under controlled interventions.<n>Results show that CMI achieves a stronger balance between answer quality and robustness to misleading memory, suggesting that reliable long-term memory requires selecting context based on causal usefulness rather than relevance alone.
  arXiv  Detail & Related papers  (2026-05-17T20:21:55Z)
• RecMem: Recurrence-based Memory Consolidation for Efficient and Effective Long-Running LLM Agents [16.625864201773343]
  RecMem stores incoming interactions in a subconscious memory layer and encodes them using lightweight embedding models for retrieval.<n>Experiments show that RecMem reduces the memory construction token cost of three SOTA memory systems by up to 87% while exceeding their accuracy.
  arXiv  Detail & Related papers  (2026-05-15T15:17:36Z)
• MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search [59.98962867649009]
  We propose MemSearch-o1, an agentic search framework built on reasoning-aligned memory growth and retracing.<n>MemSearch-o1 grows fine-grained memory fragments from memory seed tokens from queries, then retraces and deeply refines the memory via a contribution function, and finally reorganizes a globally connected memory path.
  arXiv  Detail & Related papers  (2026-04-19T05:35:06Z)
• MemMA: Coordinating the Memory Cycle through Multi-Agent Reasoning and In-Situ Self-Evolution [52.29303869050117]
  Memory-augmented LLM agents maintain external memory banks to support long-horizon interaction.<n>MemMA is a plug-and-play multi-agent framework that coordinates the memory cycle along both the forward and backward paths.
  arXiv  Detail & Related papers  (2026-03-19T10:15:59Z)
• MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks [55.145729491377374]
  Existing evaluations of agents with memory typically assess memorization and action in isolation.<n>We introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops.<n> MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning.
  arXiv  Detail & Related papers  (2026-02-18T09:49:14Z)
• Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents [57.38404718635204]
  Large language model (LLM) agents face fundamental limitations in long-horizon reasoning due to finite context windows.<n>Existing methods typically handle long-term memory (LTM) and short-term memory (STM) as separate components.<n>We propose Agentic Memory (AgeMem), a unified framework that integrates LTM and STM management directly into the agent's policy.
  arXiv  Detail & Related papers  (2026-01-05T08:24:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.