Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigation

• URL: http://arxiv.org/abs/2510.08553v1
• Date: Thu, 09 Oct 2025 17:58:01 GMT
• Title: Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigation
• Abstract summary: Vision-and-Language Navigation (VLN) requires agents to follow natural language instructions through environments.<n>Existing approaches for memory-persistent VLN face critical limitations.<n>We present Memoir, which employs imagination as a retrieval mechanism grounded by explicit memory.
• Score: 8.14424628742372
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-and-Language Navigation (VLN) requires agents to follow natural language instructions through environments, with memory-persistent variants demanding progressive improvement through accumulated experience. Existing approaches for memory-persistent VLN face critical limitations: they lack effective memory access mechanisms, instead relying on entire memory incorporation or fixed-horizon lookup, and predominantly store only environmental observations while neglecting navigation behavioral patterns that encode valuable decision-making strategies. We present Memoir, which employs imagination as a retrieval mechanism grounded by explicit memory: a world model imagines future navigation states as queries to selectively retrieve relevant environmental observations and behavioral histories. The approach comprises: 1) a language-conditioned world model that imagines future states serving dual purposes: encoding experiences for storage and generating retrieval queries; 2) Hybrid Viewpoint-Level Memory that anchors both observations and behavioral patterns to viewpoints, enabling hybrid retrieval; and 3) an experience-augmented navigation model that integrates retrieved knowledge through specialized encoders. Extensive evaluation across diverse memory-persistent VLN benchmarks with 10 distinctive testing scenarios demonstrates Memoir's effectiveness: significant improvements across all scenarios, with 5.4% SPL gains on IR2R over the best memory-persistent baseline, accompanied by 8.3x training speedup and 74% inference memory reduction. The results validate that predictive retrieval of both environmental and behavioral memories enables more effective navigation, with analysis indicating substantial headroom (73.3% vs 93.4% upper bound) for this imagination-guided paradigm. Code at https://github.com/xyz9911/Memoir.

Related papers

• Drive by Hindsight and Foresight: Tool-Grounded Synergistic Reasoning over Hierarchical Memory for Autonomous Driving [0.8419406375517101]
  VLMs have shown promise for autonomous driving yet still suffer from weak-temporal perception, and limited generalization.<n>Recent methods improve reasoning and decision-making through CoT explanations, retrieval-augmented generation or the static injection of tool outputs.<n>We present to our knowledge, the first synergistic framework that tightly couples hierarchical memory with proactive tool invocation in a closed loop.
  arXiv  Detail & Related papers  (2026-09-08T03:59:10Z)
• WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction [72.1620416874118]
  Multimodal large language models are increasingly deployed as long-horizon agents.<n>Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions.<n>We formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle.
  arXiv  Detail & Related papers  (2026-05-28T04:27:20Z)
• Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory [53.8372503928207]
  Large language model (LLM) agents require long-term user memory for consistent personalization.<n>Existing memory systems mainly rely on static, hand-crafted update rules.<n>We introduce MemCoE, a cognition-inspired two-stage optimization framework that learns how memory should be organized and what information to update.
  arXiv  Detail & Related papers  (2026-05-01T14:45:20Z)
• LMEB: Long-horizon Memory Embedding Benchmark [49.57481835614834]
  We introduce the Long-horizon Memory Embedding Benchmark (LMEB), a comprehensive framework that evaluates embedding models' capabilities.<n>LMEB spans 22 datasets and 193 zero-shot retrieval tasks across 4 memory types.<n>We evaluate 15 widely used embedding models, ranging from hundreds of millions to ten billion parameters.
  arXiv  Detail & Related papers  (2026-03-13T02:09:57Z)
• Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval [59.295767860331004]
  RF-Mem is a familiarity uncertainty-guided dual-path memory retriever.<n>It embeds human-like dual-process recognition into the retriever.<n>It consistently outperforms one-shot retrieval and full-context reasoning under fixed budget and latency constraints.
  arXiv  Detail & Related papers  (2026-03-10T06:31:44Z)
• Improving MLLMs in Embodied Exploration and Question Answering with Human-Inspired Memory Modeling [7.901667874113712]
  We propose a non-parametric memory framework that disentangles episodic and semantic memory for embodied exploration and question answering.<n>Our retrieval-first, reasoning-assisted paradigm recalls episodic experiences via semantic similarity and verifies them through visual reasoning.
  arXiv  Detail & Related papers  (2026-02-17T11:41:28Z)
• REMem: Reasoning with Episodic Memory in Language Agent [32.63834745610879]
  We present REMem, a framework for constructing and reasoning with episodic memory.<n>We show that REMem substantially outperforms state-of-the-temporal-art memory systems such as Mem0 and HippoRAG 2.<n>REMem also demonstrates more robust refusal behavior for unanswerable questions.
  arXiv  Detail & Related papers  (2026-02-13T23:54:55Z)
• Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey [211.01908189012184]
  Memory, with hundreds of papers released this year, emerges as the critical solution to fill the utility gap.<n>We provide a unified view of foundation agent memory along three dimensions.<n>We then analyze how memory is instantiated and operated under different agent topologies.
  arXiv  Detail & Related papers  (2026-01-14T07:38:38Z)
• Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution [52.76038908826961]
  We propose $textbfReMe$ ($textitRemember Me, Refine Me$) to bridge the gap between static storage and dynamic reasoning.<n>ReMe innovates across the memory lifecycle via three mechanisms: $textitmulti-faceted distillation$, which extracts fine-grained experiences.<n>Experiments on BFCL-V3 and AppWorld demonstrate that ReMe establishes a new state-of-the-art in agent memory system.
  arXiv  Detail & Related papers  (2025-12-11T14:40:01Z)
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory [89.65731902036669]
  Evo-Memory is a streaming benchmark and framework for evaluating self-evolving memory in large language model (LLM) agents.<n>We evaluate over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets.
  arXiv  Detail & Related papers  (2025-11-25T21:08:07Z)
• Evaluating Long-Term Memory for Long-Context Question Answering [100.1267054069757]
  We present a systematic evaluation of memory-augmented methods using LoCoMo, a benchmark of synthetic long-context dialogues annotated for question-answering tasks.<n>Our findings show that memory-augmented approaches reduce token usage by over 90% while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2025-10-27T18:03:50Z)
• Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation [11.819481846962447]
  We investigate how agents built on pretrained large language models can learn target classification functions from labeled examples without parameter updates.<n>Our framework uses episodic memory to store instance-level critiques and distill these into reusable, task-level guidance.<n>Our findings highlight the promise of memory-driven, reflective learning for building more adaptive and interpretable LLM agents.
  arXiv  Detail & Related papers  (2025-10-22T17:58:03Z)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents [57.1835920227202]
  We propose MemGen, a dynamic generative memory framework that equips agents with a human-esque cognitive faculty.<n>MemGen enables agents to recall and augment latent memory throughout reasoning, producing a tightly interwoven cycle of memory and cognition.
  arXiv  Detail & Related papers  (2025-09-29T12:33:13Z)
• EXPEREPAIR: Dual-Memory Enhanced LLM-based Repository-Level Program Repair [13.512456346600477]
  We propose ExpeRepair, a novel approach to automatically repairing software issues.<n>It learns from historical repair experiences through dual-channel knowledge accumulation.<n>It achieves a pass@1 score of 49.3% with Claude 3.7 Sonnet, outperforming all state-of-the-art open-source methods.
  arXiv  Detail & Related papers  (2025-06-12T08:39:27Z)
• Memory-enhanced Retrieval Augmentation for Long Video Understanding [91.7163732531159]
  We introduce a novel memory-enhanced RAG-based approach called MemVid.<n>Our approach operates in four basic steps: 1) memorizing holistic video information, 2) reasoning about the task's information needs based on memory, 3) retrieving critical moments based on the information needs, and 4) focusing on the retrieved moments to produce the final answer.<n>MemVid demonstrates superior efficiency and effectiveness compared to both LVLMs and RAG methods.
  arXiv  Detail & Related papers  (2025-03-12T08:23:32Z)
• In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents [70.12342024019044]
  Large Language Models (LLMs) have made significant progress in open-ended dialogue, yet their inability to retain and retrieve relevant information limits their effectiveness.<n>We propose Reflective Memory Management (RMM), a novel mechanism for long-term dialogue agents, integrating forward- and backward-looking reflections.<n>RMM shows more than 10% accuracy improvement over the baseline without memory management on the LongMemEval dataset.
  arXiv  Detail & Related papers  (2025-03-11T04:15:52Z)
• Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning [64.93848182403116]
  Current deep-learning memory models struggle in reinforcement learning environments that are partially observable and long-term. We introduce the Stable Hadamard Memory, a novel memory model for reinforcement learning agents. Our approach significantly outperforms state-of-the-art memory-based methods on challenging partially observable benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T03:50:17Z)
• MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation [15.64077949677469]
  We present a novel Memory-Augmented Dialogue Benchmark (MADail-Bench) to evaluate the effectiveness of memory-augmented dialogue systems (MADS) The benchmark assesses two tasks separately: memory retrieval and memory recognition with the incorporation of both passive and proactive memory recall data. Results from cutting-edge embedding models and large language models on this benchmark indicate the potential for further advancement.
  arXiv  Detail & Related papers  (2024-09-23T17:38:41Z)
• Black-box Unsupervised Domain Adaptation with Bi-directional Atkinson-Shiffrin Memory [59.51934126717572]
  Black-box unsupervised domain adaptation (UDA) learns with source predictions of target data without accessing either source data or source models during training. We propose BiMem, a bi-directional memorization mechanism that learns to remember useful and representative information to correct noisy pseudo labels on the fly. BiMem achieves superior domain adaptation performance consistently across various visual recognition tasks such as image classification, semantic segmentation and object detection.
  arXiv  Detail & Related papers  (2023-08-25T08:06:48Z)
• Saliency-Augmented Memory Completion for Continual Learning [8.243137410556495]
  How to forget is a problem continual learning must address. Our paper proposes a new saliency-augmented memory completion framework for continual learning.
  arXiv  Detail & Related papers  (2022-12-26T18:06:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.