Mitigating Over-Personalization in LLMs via Structured Memory

• URL: http://arxiv.org/abs/2608.08300v1
• Date: Sat, 08 Aug 2026 19:23:12 GMT
• Title: Mitigating Over-Personalization in LLMs via Structured Memory
• Abstract summary: We study two failure modes in memory-augmented LLMs: cross-domain leakage and memory-induced sycophancy.<n>We apply a simple inference-time modification to how memories are presented to the model, without changing the model or the memory contents.
• Score: 8.188066927114779
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conversational assistants increasingly rely on persistent long-term memory to personalize responses across sessions. However, when stored user information is reintroduced into the model context, it can also influence responses in inappropriate or unrelated settings. We study two such failure modes in memory-augmented LLMs: cross-domain leakage, where memories from one life domain affect responses in another, and memory-induced sycophancy, where stored user beliefs make models more likely to agree with the user rather than respond truthfully. We apply a simple inference-time modification to how memories are presented to the model, without changing the model or the memory contents. Across seven models on PersistBench, we compare the commonly used all-in context format, where memories are injected as an unstructured list, with structured formats that partition memories by domain. This simple modification consistently reduces cross-domain leakage while preserving utility, with our strongest method reducing leakage by $8.8\%$ on average relative to the baseline.

Related papers

• MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use [30.64715962774191]
  Existing memory benchmarks mainly evaluate whether information is correctly extracted, stored, and retrieved.<n>We identify memory-induced cognitive traps: even faithfully recorded and semantically relevant memories can distort model reasoning.<n>We propose AdaptiveMem, a simple yet effective inference-time method that instructs LLMs to avoid memory traps.
  arXiv  Detail & Related papers  (2026-08-20T16:00:17Z)
• Beyond Similarity: Trustworthy Memory Search for Personal AI Agents [25.265839311088516]
  We study memory search as a trust boundary in personal AI agents.<n>MemGate is a lightweight and deployable memory plug-in for trustworthy memory search.
  arXiv  Detail & Related papers  (2026-06-04T11:54:29Z)
• Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents [51.30250860677378]
  We propose MRAgent, a framework that combines an associative memory graph with an active reconstruction mechanism.<n>Running on a Cue-Tag-Content graph, our active reconstruction mechanism integrates reasoning directly into memory access.<n>Experiments on the LoCoMo benchmark and LongMemEval benchmark demonstrate significant improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T11:29:46Z)
• MemGuard: Preventing Memory Contamination in Long-Term Memory-Augmented Large Language Models [56.31411457917676]
  We introduce MemGuard, a type-aware memory framework that preserves functional memory boundaries during memory construction and retrieval.<n>Across hallucination and long-horizon conversation benchmarks, MemGuard improves memory reliability by up to 28.27% while retrieving up to 5.8x fewer memory tokens than prior methods.
  arXiv  Detail & Related papers  (2026-05-27T06:04:19Z)
• Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents [20.22872890297194]
  Existing large language model (LLM) based memory systems apply universal, static policies that overlook a fundamental reality: the contexts that are worth storing in memory are different across users.<n>We introduce PerMemBench, the first benchmark for evaluating personalized memory systems.<n>We propose session level storage gating, a lightweight framework that selectively bypasses memory operations for transient sessions.
  arXiv  Detail & Related papers  (2026-05-25T07:48:33Z)
• Causal Intervention-Based Memory Selection for Long-Horizon LLM Agents [0.33842793760651557]
  We propose a causal memory-selection technique that estimates how candidate memories affect the model's answer under controlled interventions.<n>Results show that CMI achieves a stronger balance between answer quality and robustness to misleading memory, suggesting that reliable long-term memory requires selecting context based on causal usefulness rather than relevance alone.
  arXiv  Detail & Related papers  (2026-05-17T20:21:55Z)
• Memory Transfer Learning: How Memories are Transferred Across Domains in Coding Agents [56.078070829341534]
  Memory-based self-evolution has emerged as a promising paradigm for coding agents.<n>We investigate textbfMemory Transfer Learning (MTL) by harnessing a unified memory pool from heterogeneous domains.
  arXiv  Detail & Related papers  (2026-04-15T15:50:29Z)
• Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval [59.295767860331004]
  RF-Mem is a familiarity uncertainty-guided dual-path memory retriever.<n>It embeds human-like dual-process recognition into the retriever.<n>It consistently outperforms one-shot retrieval and full-context reasoning under fixed budget and latency constraints.
  arXiv  Detail & Related papers  (2026-03-10T06:31:44Z)
• Pre-Storage Reasoning for Episodic Memory: Shifting Inference Burden to Memory for Personalized Dialogue [13.558061425427688]
  PREMem is a novel approach that shifts complex reasoning processes from inference to memory construction.<n>It creates enriched representations while reducing computational demands during interactions.<n> Experiments show significant performance improvements across all model sizes.
  arXiv  Detail & Related papers  (2025-09-13T15:18:08Z)
• ArcMemo: Abstract Reasoning Composition with Lifelong LLM Memory [21.4675019810992]
  Concept-level memory is reusable, modular abstractions distilled from solution traces and stored in natural language.<n>We evaluate on ARC-AGI, a benchmark that stresses compositional generalization and abstract reasoning.<n>We find abstract concepts to be the most consistent memory design, outscoring the baseline at all tested inference compute scales.
  arXiv  Detail & Related papers  (2025-09-04T17:54:19Z)
• A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning [56.450090618578]
  Class-Incremental Learning (CIL) aims to train a model with limited memory size to meet this requirement. We show that when counting the model size into the total budget and comparing methods with aligned memory size, saving models do not consistently work. We propose a simple yet effective baseline, denoted as MEMO for Memory-efficient Expandable MOdel.
  arXiv  Detail & Related papers  (2022-05-26T08:24:01Z)
• LaMemo: Language Modeling with Look-Ahead Memory [50.6248714811912]
  We propose Look-Ahead Memory (LaMemo) that enhances the recurrence memory by incrementally attending to the right-side tokens. LaMemo embraces bi-directional attention and segment recurrence with an additional overhead only linearly proportional to the memory length. Experiments on widely used language modeling benchmarks demonstrate its superiority over the baselines equipped with different types of memory.
  arXiv  Detail & Related papers  (2022-04-15T06:11:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.