Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval

• URL: http://arxiv.org/abs/2603.09250v1
• Date: Tue, 10 Mar 2026 06:31:44 GMT
• Title: Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval
• Abstract summary: RF-Mem is a familiarity uncertainty-guided dual-path memory retriever.<n>It embeds human-like dual-process recognition into the retriever.<n>It consistently outperforms one-shot retrieval and full-context reasoning under fixed budget and latency constraints.
• Score: 59.295767860331004
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized large language models (LLMs) rely on memory retrieval to incorporate user-specific histories, preferences, and contexts. Existing approaches either overload the LLM by feeding all the user's past memory into the prompt, which is costly and unscalable, or simplify retrieval into a one-shot similarity search, which captures only surface matches. Cognitive science, however, shows that human memory operates through a dual process: Familiarity, offering fast but coarse recognition, and Recollection, enabling deliberate, chain-like reconstruction for deeply recovering episodic content. Current systems lack both the ability to perform recollection retrieval and mechanisms to adaptively switch between the dual retrieval paths, leading to either insufficient recall or the inclusion of noise. To address this, we propose RF-Mem (Recollection-Familiarity Memory Retrieval), a familiarity uncertainty-guided dual-path memory retriever. RF-Mem measures the familiarity signal through the mean score and entropy. High familiarity leads to the direct top-K Familiarity retrieval path, while low familiarity activates the Recollection path. In the Recollection path, the system clusters candidate memories and applies alpha-mix with the query to iteratively expand evidence in embedding space, simulating deliberate contextual reconstruction. This design embeds human-like dual-process recognition into the retriever, avoiding full-context overhead and enabling scalable, adaptive personalization. Experiments across three benchmarks and corpus scales demonstrate that RF-Mem consistently outperforms both one-shot retrieval and full-context reasoning under fixed budget and latency constraints. Our code can be found in the Reproducibility Statement.

Related papers

• Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering [42.31517185994942]
  RAG enables large language models to answer questions by accessing external knowledge.<n>Existing methods suffer from two key challenges: fragmented evidence scattered across temporally distant sessions, and noisy content within retrieved sessions that triggers the lost-in-the-middle effect.<n>We propose MemLoc, a unified Retrieve-Localize-Generate framework for long-term conversational memory QA.
  arXiv  Detail & Related papers  (2026-09-07T06:32:12Z)
• InjecMEM: Memory Injection Attack on LLM Agent Memory Systems [31.104106193922956]
  InjecMEM is a novel memory injection attack paradigm.<n>We craft the injection with a retriever-agnostic anchor and an adversarial command.<n>We evaluate it across multiple memory systems and backbone models.
  arXiv  Detail & Related papers  (2026-08-24T16:37:50Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents [51.30250860677378]
  We propose MRAgent, a framework that combines an associative memory graph with an active reconstruction mechanism.<n>Running on a Cue-Tag-Content graph, our active reconstruction mechanism integrates reasoning directly into memory access.<n>Experiments on the LoCoMo benchmark and LongMemEval benchmark demonstrate significant improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T11:29:46Z)
• Rethinking Memory as Continuously Evolving Connectivity [63.58181883607843]
  Existing memory-augmented LLM agents treat memory as a static repository with pre-defined representations and fixed retrieval pipelines.<n>We propose FluxMem, a connectivity-evolving memory framework that models memory as a heterogeneous graph and progressively refines its topology through three stages.<n>Across three fundamentally distinct benchmarks including LoCoMo, Mind2Web, and GAIA, FluxMem achieves consistent state-of-the-art performance, demonstrating strong adaptation and generalization in complex agentic environments.
  arXiv  Detail & Related papers  (2026-05-27T17:35:34Z)
• RecMem: Recurrence-based Memory Consolidation for Efficient and Effective Long-Running LLM Agents [16.625864201773343]
  RecMem stores incoming interactions in a subconscious memory layer and encodes them using lightweight embedding models for retrieval.<n>Experiments show that RecMem reduces the memory construction token cost of three SOTA memory systems by up to 87% while exceeding their accuracy.
  arXiv  Detail & Related papers  (2026-05-15T15:17:36Z)
• REMem: Reasoning with Episodic Memory in Language Agent [32.63834745610879]
  We present REMem, a framework for constructing and reasoning with episodic memory.<n>We show that REMem substantially outperforms state-of-the-temporal-art memory systems such as Mem0 and HippoRAG 2.<n>REMem also demonstrates more robust refusal behavior for unanswerable questions.
  arXiv  Detail & Related papers  (2026-02-13T23:54:55Z)
• AMA: Adaptive Memory via Multi-Agent Collaboration [54.490349689939166]
  We propose Adaptive Memory via Multi-Agent Collaboration (AMA), a novel framework that leverages coordinated agents to manage memory across multiple granularities.<n>AMA significantly outperforms state-of-the-art baselines while reducing token consumption by approximately 80% compared to full-context methods.
  arXiv  Detail & Related papers  (2026-01-28T08:09:49Z)
• Agentic Learner with Grow-and-Refine Multimodal Semantic Memory [50.81667005063605]
  ViLoMem is a dual-stream memory framework that constructs compact, schema-based memory.<n>It encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences.
  arXiv  Detail & Related papers  (2025-11-26T18:55:08Z)
• Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents [33.617262543252494]
  We present ReMemR1, a memory-augmented agent with callback-enhanced memory that allows selective retrieval from the entire memory history.<n>We also propose Reinforcement Learning with Multi-Level Rewards (RLMLR), which combines final-answer rewards with dense, step-level signals that guide effective memory use.
  arXiv  Detail & Related papers  (2025-09-27T01:36:46Z)
• From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents [79.87304940020256]
  Large Language Models (LLMs) have been widely adopted in conversational agents.<n>MemGAS is a framework that enhances memory consolidation by constructing multi-granularity association, adaptive selection, and retrieval.<n> Experiments on four long-term memory benchmarks demonstrate that MemGAS outperforms state-of-the-art methods on both question answer and retrieval tasks.
  arXiv  Detail & Related papers  (2025-05-26T06:13:07Z)
• Adversarially Diversified Rehearsal Memory (ADRM): Mitigating Memory Overfitting Challenge in Continual Learning [0.0]
  Continual learning focuses on learning non-stationary data distribution without forgetting previous knowledge. Rehearsal-based approaches are commonly used to combat catastrophic forgetting. We introduce the Adversarially Diversified Rehearsal Memory to address the memory overfitting challenge.
  arXiv  Detail & Related papers  (2024-05-20T06:56:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.