HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory System

• URL: http://arxiv.org/abs/2603.14807v1
• Date: Mon, 16 Mar 2026 04:23:29 GMT
• Title: HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory System
• Abstract summary: We propose HiMeVLN, which incorporates a Hierarchical Memory System into a multimodal large model to enhance visual perception and long-term localization.<n>Experiments in both simulated and real-world environments demonstrate that HiMeVLN achieves nearly twice the performance of the open-source method.
• Score: 12.907741491900731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based agents have demonstrated impressive zero-shot performance in vision-language navigation (VLN) tasks. However, most zero-shot methods primarily rely on closed-source LLMs as navigators, which face challenges related to high token costs and potential data leakage risks. Recent efforts have attempted to address this by using open-source LLMs combined with a spatiotemporal CoT framework, but they still fall far short compared to closed-source models. In this work, we identify a critical issue, Navigation Amnesia, through a detailed analysis of the navigation process. This issue leads to navigation failures and amplifies the gap between open-source and closed-source methods. To address this, we propose HiMemVLN, which incorporates a Hierarchical Memory System into a multimodal large model to enhance visual perception recall and long-term localization, mitigating the amnesia issue and improving the agent's navigation performance. Extensive experiments in both simulated and real-world environments demonstrate that HiMemVLN achieves nearly twice the performance of the open-source state-of-the-art method. The code is available at https://github.com/lvkailin0118/HiMemVLN.

Related papers

• CMMR-VLN: Vision-and-Language Navigation via Continual Multimodal Memory Retrieval [8.022522932302998]
  Vision-and-language navigation (VLN) lacks the ability to selectively recall and use relevant priori experiences to help navigation tasks.<n>We propose CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities.
  arXiv  Detail & Related papers  (2026-03-09T06:02:50Z)
• SFCo-Nav: Efficient Zero-Shot Visual Language Navigation via Collaboration of Slow LLM and Fast Attributed Graph Alignment [7.6686624731596895]
  We present SFCo-Nav, an efficient zero-shot visual language navigation framework.<n>It integrates three key modules: 1) a slow LLM-based planner that produces a strategic chain of subgoals, each linked to an imagined object graph; 2) a fast reactive navigator for real-time object graph construction and subgoal execution; and 3) a lightweight asynchronous slow-fast bridge aligns advanced structured, attributed imagined and perceived graphs to estimate navigation confidence.<n>It matches or exceeds prior state-of-the-art zero-shot VLN success rates while cutting total token consumption per trajectory by over 50% and running more than 3.5
  arXiv  Detail & Related papers  (2026-03-02T05:42:12Z)
• VidLaDA: Bidirectional Diffusion Large Language Models for Efficient Video Understanding [52.69880888587866]
  Current Video Large Language Models (Video LLMs) typically encode frames via a encoder vision and employ an autoregressive (AR) LLM for understanding and generation.<n>We propose VidLaDA, a Diffusion Video LLM based on Language Models (DLMs) that leverages bidirectional attention to unlock comprehensive modeling and decode tokens in parallel.<n>Experiments show VidLaDA rivals state-of-the-art AR baselines and outperforms DLM baselines, with MARS-Cache delivering over 12x speedup without compromising accuracy.
  arXiv  Detail & Related papers  (2026-01-25T15:02:01Z)
• Agentic Learner with Grow-and-Refine Multimodal Semantic Memory [50.81667005063605]
  ViLoMem is a dual-stream memory framework that constructs compact, schema-based memory.<n>It encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences.
  arXiv  Detail & Related papers  (2025-11-26T18:55:08Z)
• DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation [55.888688171010365]
  DORAEMON is a cognitive-inspired framework consisting of Ventral and Dorsal Streams that mimics human navigation capabilities.<n>We evaluate DORAEMON on the HM3D, MP3D and GOAT datasets, where it achieves state-of-the-art performance on both success rate (SR) and success weighted by path length (SPL) metrics.
  arXiv  Detail & Related papers  (2025-05-28T04:46:13Z)
• Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference [82.557441484339]
  We propose a method to detect the knowledge boundary of Vision Large Language Models (VLLMs)<n> Experimental results on various types of Visual Question Answering datasets show that our method successfully depicts a VLLM's knowledge boundary.
  arXiv  Detail & Related papers  (2025-02-25T09:32:08Z)
• Navigation-GPT: A Robust and Adaptive Framework Utilizing Large Language Models for Navigation Applications [6.990141986853289]
  Existing navigation decision support systems often perform poorly when handling non-predefined scenarios.<n>This research proposes a dual-core framework for LLM applications to address this issue.
  arXiv  Detail & Related papers  (2025-02-23T01:41:58Z)
• Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs [41.90732562248243]
  Vision-and-Language Navigation (VLN) tasks require an agent to follow textual instructions to navigate through 3D environments.<n>Recent methods try to utilize closed-source large language models (LLMs) to solve VLN tasks in zero-shot manners.<n>We introduce Open-Nav, a novel study that explores open-source LLMs for zero-shot VLN in the continuous environment.
  arXiv  Detail & Related papers  (2024-09-27T14:47:18Z)
• NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning [97.88246428240872]
  Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.<n>Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability.<n>This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision.
  arXiv  Detail & Related papers  (2024-03-12T07:27:02Z)
• Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation [79.1669476932147]
  Vision-and-Language Navigation (VLN) is a task that an agent is required to follow a language instruction to navigate to the goal position. Recent Transformer-based VLN methods have made great progress benefiting from the direct connections between visual observations and the language instruction. We introduce Multimodal Transformer with Variable-length Memory (MTVM) for visually-grounded natural language navigation.
  arXiv  Detail & Related papers  (2021-11-10T16:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.