HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation

• URL: http://arxiv.org/abs/2607.29600v1
• Date: Fri, 31 Jul 2026 16:32:38 GMT
• Title: HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation
• Abstract summary: Vision-and-language navigation (VLN) enables robots to follow instructions in previously unseen environments.<n>Ham-VLN is a decision-coupled, agent-authored memory that equips the robot with a persistent, depth-grounded world graph.<n>Ham-VLN achieves 61.0% Success Rate (SR) on VLN-CE R2R, 52.7% SR on VLN-CE RxR, and 79.7% SR on HM3D-v2 ObjectNav without any training.
• Score: 19.550897283368894
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-and-language navigation (VLN) enables robots to follow instructions in previously unseen environments. Recently, a training-free paradigm has emerged: the robot queries a multimodal LLM to understand its observations and plan the next action. However, long-horizon navigation based on either image streams or dense map inevitably introduces a growing memory and reasoning bottleneck. We present HAM-VLN, a decision-coupled, agent-authored memory that equips the robot with a persistent, depth-grounded world graph. In the same model call used to select the next action, HAM-VLN also records semantic and reflective information---including room type, objects, navigation progress, and failure notes. Recent waypoints remain verbatim within a bounded window, while older history re-enters the context only through retrieval scored by relevance, recency, and salience, together with one-hop topological expansion. This design requires no additional LLM calls beyond the per-waypoint decision. Compared to previous methods, HAM-VLN not only improves various navigation metrics but also reduces the context length by more than 65%. Specifically, HAM-VLN achieves 61.0% Success Rate (SR) on VLN-CE R2R, 52.7% SR on VLN-CE RxR, and 79.7% SR on HM3D-v2 ObjectNav without any training.

Related papers

• AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness [5.851139427049915]
  Zero-shot vision-and-language navigation in continuous environments (VLN-CE) has recently become feasible with large vision-language models (VLMs)<n>In this paper, we rethink zero-shot VLN-CE as an agentic interface between the VLM and the environment.<n>We present AgenticNav, a lightweight navigation harness that exposes action, depth, and memory as callable tools.
  arXiv  Detail & Related papers  (2026-06-09T08:43:05Z)
• IntentNav: Learning Spatial-Visual Object Navigation from Human Demonstrations [58.921283404811085]
  We propose a framework that learns human-like ObjectNav policies from human demonstrations.<n>IntentNav achieves state-of-the-art performance on the MP3D, HM3D-v1 and HM3D-v2 ObjectNav benchmarks.
  arXiv  Detail & Related papers  (2026-06-06T07:45:19Z)
• EvoMemNav: Efficient Self-Evolving Fine-Grained Memory for Zero-Shot Embodied Navigation [81.54723508469617]
  EvoMemNav is an efficient, self-evolving, fine-grained memory framework for zero-shot embodied navigation.<n>VSMGraph organizes views with semantic cues and topological relations into a room-view-object hierarchy.<n>Experiments on GOAT-Bench and HM3D across object, text-description, and image-goal modalities show consistent gains in SR/SPL.
  arXiv  Detail & Related papers  (2026-06-02T11:27:44Z)
• Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation [60.07205156194741]
  Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen.<n>We present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families and to four heterogeneous real robots.
  arXiv  Detail & Related papers  (2026-05-26T18:52:04Z)
• ReMemNav: A Rethinking and Memory-Augmented Framework for Zero-Shot Object Navigation [35.416693138335354]
  Zero-shot object navigation requires agents to locate unseen target objects in unfamiliar environments.<n>Recent advancements in vision-language models provide promising commonsense reasoning capabilities for this task.<n>We propose a novel hierarchical navigation framework named ReMemNav, which seamlessly integrates panoramic semantic priors and episodic memory.
  arXiv  Detail & Related papers  (2026-03-25T09:07:32Z)
• GoalVLM: VLM-driven Object Goal Navigation for Multi-Agent System [0.0]
  We present GoalVLM, a cooperative multi-agent framework for zero-shot, open-vocabulary object navigation.<n>GoalVLM integrates a Vision-Language Model (VLM) directly into the decision loop, SAM3 for text-prompted detection and segmentation, and SpaceOM for spatial reasoning.<n>We evaluate GoalVLM on GOAT-Bench val_unseen, where each episode requires navigating to a chain of 5-7 open-vocabulary targets.
  arXiv  Detail & Related papers  (2026-03-18T18:59:33Z)
• History-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigation [5.343932820859596]
  This paper introduces a novel zero-shot ObjectNav framework that pioneers the use of dynamic, history-aware prompting.<n>Our core innovation lies in providing the VLM with action history context, enabling it to generate semantic guidance scores for navigation actions.<n>We also introduce a VLM-assisted waypoint generation mechanism for refining the final approach to detected objects.
  arXiv  Detail & Related papers  (2025-06-19T21:50:16Z)
• DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation [55.888688171010365]
  DORAEMON is a cognitive-inspired framework consisting of Ventral and Dorsal Streams that mimics human navigation capabilities.<n>We evaluate DORAEMON on the HM3D, MP3D and GOAT datasets, where it achieves state-of-the-art performance on both success rate (SR) and success weighted by path length (SPL) metrics.
  arXiv  Detail & Related papers  (2025-05-28T04:46:13Z)
• NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning [97.88246428240872]
  Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.<n>Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability.<n>This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision.
  arXiv  Detail & Related papers  (2024-03-12T07:27:02Z)
• Structured Scene Memory for Vision-Language Navigation [155.63025602722712]
  We propose a crucial architecture for vision-language navigation (VLN) It is compartmentalized enough to accurately memorize the percepts during navigation. It also serves as a structured scene representation, which captures and disentangles visual and geometric cues in the environment.
  arXiv  Detail & Related papers  (2021-03-05T03:41:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.