When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents

• URL: http://arxiv.org/abs/2608.04574v1
• Date: Wed, 05 Aug 2026 08:04:26 GMT
• Title: When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents
• Abstract summary: Memory-augmented VLM agents act on persistent spatial knowledge, yet that knowledge silently goes stale as the environment changes.<n>We ask what happens when an agent must reconcile a confident memory claim with a contradicting observation, and whether current models can catch the conflict before it becomes a safety-relevant mistake.
• Score: 1.095093532533584
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Memory-augmented VLM agents act on persistent spatial knowledge, yet that knowledge silently goes stale as the environment changes. We ask what happens when an agent must reconcile a confident memory claim with a contradicting observation, and whether current models can catch the conflict before it becomes a safety-relevant mistake. Using a dynamic FrozenLake testbed, we pair a staleness-detection task with a downstream navigation task across three closed-source models and three open-weight VLMs under both text and image inputs (1,800 detection runs, and 12,000 text-mode navigation episodes over four LLM navigators at a shared 50-seed scale). Three findings emerge. First, text solvability does not imply visual grounding: models that flag stale entries reliably from text nonetheless span vision F1 from 0.887 down to 0.067 on the identical grids, and the weakest keeps making fluent, confident decisions that ignore the image. Second, consuming stale memory without an audit is a safety liability: in our primary GPT-4o setting, an agent that trusts raw memory dies more than twice as often as the same agent given no memory at all. Third, auditing helps but does not close the gap: a transparent read-time filter removes much of the safety cost in text mode, yet even oracle stale labels bring no further significant gain on the current grid size, and when visual auditing is unreliable, filtering yields no consistent benefit. Together these results frame spatial-memory staleness as a safety failure mode and isolate reliable visual grounding and action selection under memory--observation conflict as the central open challenges for memory-augmented agents.

Related papers

• The Memory Trust Gap: Capability-Dependent Failures in Persistent-Memory Agents [0.0]
  Persistent memory supports personalized agents, but a stale stored fact can override current authoritative evidence without warning.<n>We study when this harm begins as model capability changes.
  arXiv  Detail & Related papers  (2026-09-01T20:35:00Z)
• Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner [55.80248281761638]
  multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning.<n>We propose DG-Mem, a dual-grained agentic memory framework that augments a frozen MLLM with a non- externally stored memory built once from training-time rollouts and consulted read-only at test time.
  arXiv  Detail & Related papers  (2026-08-24T13:56:01Z)
• Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents [72.3752981234916]
  Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks.<n>We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process.<n>MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5-20%.
  arXiv  Detail & Related papers  (2026-07-15T08:32:40Z)
• Beyond Similarity: Trustworthy Memory Search for Personal AI Agents [25.265839311088516]
  We study memory search as a trust boundary in personal AI agents.<n>MemGate is a lightweight and deployable memory plug-in for trustworthy memory search.
  arXiv  Detail & Related papers  (2026-06-04T11:54:29Z)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction [72.1620416874118]
  Multimodal large language models are increasingly deployed as long-horizon agents.<n>Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions.<n>We formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle.
  arXiv  Detail & Related papers  (2026-05-28T04:27:20Z)
• Learning When to Remember: Risk-Sensitive Contextual Bandits for Abstention-Aware Memory Retrieval in LLM-Based Coding Agents [0.0]
  coding agents increasingly rely on external memory to reuse prior experience, repair traces, and repository-local operational knowledge.<n>This paper reframes issue-memory use as a selective, risk-sensitive control problem rather than a pure top-k retrieval problem.<n>We introduce RSCB-MC, a risk-sensitive contextual bandit memory controller that decides whether an agent should use no memory, inject the top resolution, summarize multiple candidates, perform high-precision or high-recall retrieval, abstain, or ask for feedback.
  arXiv  Detail & Related papers  (2026-04-30T00:32:53Z)
• MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents [78.95081012334116]
  Large Language Models (LLMs) with persistent memory enhances interaction continuity and personalization but introduces new safety risks.<n> contaminated or biased memory accumulation can trigger abnormal agent behaviors.<n>MemeEvoBench is the first benchmark evaluating memory safety in LLM agents against adversarial memory injection, noisy tool outputs, and biased feedback.
  arXiv  Detail & Related papers  (2026-04-17T07:29:52Z)
• Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers [0.42061757959666934]
  Large language model (LLM) agents increasingly operate in settings where a single context window is too small to capture what has happened.<n>Memory is what turns a stateless text generator into a genuinely adaptive agent.<n>This survey offers a structured account of how memory is designed, implemented, and evaluated in modern LLM-based agents.
  arXiv  Detail & Related papers  (2026-03-08T15:08:01Z)
• EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents [52.567469286881426]
  We introduce EMemBench, a programmatic benchmark for evaluating long-term memory of agents through interactive games.<n>Rather than using a fixed set of questions, EMemBench generates questions from each agent's own trajectory.<n>Each template computes verifiable ground truth from underlying game signals.
  arXiv  Detail & Related papers  (2026-01-23T12:09:59Z)
• Agentic Learner with Grow-and-Refine Multimodal Semantic Memory [50.81667005063605]
  ViLoMem is a dual-stream memory framework that constructs compact, schema-based memory.<n>It encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences.
  arXiv  Detail & Related papers  (2025-11-26T18:55:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.