CMMR-VLN: Vision-and-Language Navigation via Continual Multimodal Memory Retrieval

• URL: http://arxiv.org/abs/2603.07997v1
• Date: Mon, 09 Mar 2026 06:02:50 GMT
• Title: CMMR-VLN: Vision-and-Language Navigation via Continual Multimodal Memory Retrieval
• Abstract summary: Vision-and-language navigation (VLN) lacks the ability to selectively recall and use relevant priori experiences to help navigation tasks.<n>We propose CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities.
• Score: 8.022522932302998
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although large language models (LLMs) are introduced into vision-and-language navigation (VLN) to improve instruction comprehension and generalization, existing LLM- based VLN lacks the ability to selectively recall and use relevant priori experiences to help navigation tasks, limiting their performance in long-horizon and unfamiliar scenarios. In this work, we propose CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities. Specifically, the CMMR-VLN constructs a multimodal experi- ence memory indexed by panoramic visual images and salient landmarks to retrieve relevant experiences during navigation, introduces a retrieved-augmented generation pipeline to mimick how experienced human navigators leverage priori knowledge, and incorporates a reflection-based memory update strategy that selectively stores complete successful paths and the key initial mistake in failure cases. Comprehensive tests illustrate average success rate improvements of 52.9%, 20.9% and 20.9%, and 200%, 50% and 50% over the NavGPT, the MapGPT, and the DiscussNav in simulation and real tests, respectively eluci- dating the great potential of the CMMR-VLN as a backbone VLN framework.

Related papers

• HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory System [12.907741491900731]
  We propose HiMeVLN, which incorporates a Hierarchical Memory System into a multimodal large model to enhance visual perception and long-term localization.<n>Experiments in both simulated and real-world environments demonstrate that HiMeVLN achieves nearly twice the performance of the open-source method.
  arXiv  Detail & Related papers  (2026-03-16T04:23:29Z)
• VLM-Guided Experience Replay [41.08659748023147]
  We propose to use Vision-Language Models (VLMs) to guide the prioritization of experiences in the replay buffer.<n>Our key idea is to use a frozen, pre-trained VLM as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences.<n>Across scenarios, including game-playing and robotics, agents trained with our proposed prioritization method achieve 11-52% higher average success rates and improve sample efficiency by 19-45% compared to previous approaches.
  arXiv  Detail & Related papers  (2026-02-02T10:19:59Z)
• VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents [12.383467162169703]
  We introduce a unified and evaluation framework to probe MLLMs as zero-shot agents.<n>We simplify the evaluation with a highly modular and accessible design.<n>We observe that enhancing our baseline agent with Chain-of-Thought (CoT) reasoning and self-language leads to an unexpected performance decrease.
  arXiv  Detail & Related papers  (2025-12-31T13:21:21Z)
• Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory [89.65731902036669]
  Evo-Memory is a streaming benchmark and framework for evaluating self-evolving memory in large language model (LLM) agents.<n>We evaluate over ten representative memory modules and evaluate them across 10 diverse multi-turn goal-oriented and single-turn reasoning and QA datasets.
  arXiv  Detail & Related papers  (2025-11-25T21:08:07Z)
• SE-VLN: A Self-Evolving Vision-Language Navigation Framework Based on Multimodal Large Language Models [8.48034992736753]
  The SE-VLN is a self-evolving framework for vision-language navigation (VLN)<n>It consists of three core modules, i.e., a hierarchical memory module, a retrieval-augmented thought-based reasoning module, and a reflection module.<n>It achieved navigation success rates of 57% and 35.2% in unseen environments, representing absolute performance improvements of 23.9% and 15.0% over current methods on R2R and REVERSE datasets, respectively.
  arXiv  Detail & Related papers  (2025-07-17T14:13:50Z)
• DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation [55.888688171010365]
  DORAEMON is a cognitive-inspired framework consisting of Ventral and Dorsal Streams that mimics human navigation capabilities.<n>We evaluate DORAEMON on the HM3D, MP3D and GOAT datasets, where it achieves state-of-the-art performance on both success rate (SR) and success weighted by path length (SPL) metrics.
  arXiv  Detail & Related papers  (2025-05-28T04:46:13Z)
• Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Method [94.74003109176581]
  Long-Horizon Vision-Language Navigation (LH-VLN) is a novel VLN task that emphasizes long-term planning and decision consistency across consecutive subtasks.<n>Our platform, benchmark and method supply LH-VLN with a robust data generation pipeline, comprehensive model evaluation dataset, reasonable metrics, and a novel VLN model.
  arXiv  Detail & Related papers  (2024-12-12T09:08:13Z)
• NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning [97.88246428240872]
  Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.<n>Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability.<n>This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision.
  arXiv  Detail & Related papers  (2024-03-12T07:27:02Z)
• ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning [73.47165576175541]
  Two-Tower Vision-Language (VL) models have shown promising improvements on various downstream tasks. We propose ManagerTower, a novel VL model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels.
  arXiv  Detail & Related papers  (2023-05-31T18:23:57Z)
• Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation [79.1669476932147]
  Vision-and-Language Navigation (VLN) is a task that an agent is required to follow a language instruction to navigate to the goal position. Recent Transformer-based VLN methods have made great progress benefiting from the direct connections between visual observations and the language instruction. We introduce Multimodal Transformer with Variable-length Memory (MTVM) for visually-grounded natural language navigation.
  arXiv  Detail & Related papers  (2021-11-10T16:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.