NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularization

• URL: http://arxiv.org/abs/2507.10894v1
• Date: Tue, 15 Jul 2025 01:20:22 GMT
• Title: NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularization
• Authors: Zongtao He, Liuyi Wang, Lu Chen, Chengju Liu, Qijun Chen,
• Abstract summary: We propose NavComposer, a framework for automatically generating high-quality navigation instructions.<n>NavComposer explicitly decomposes semantic entities such as actions, scenes, and objects, and recomposes them into natural language instructions.<n>It operates in a data-agnostic manner, supporting adaptation to diverse navigation trajectories without domain-specific training.<n>NavInstrCritic provides a holistic evaluation of instruction quality, addressing limitations of traditional metrics that rely heavily on expert annotations.
• Score: 17.525269369227786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language-guided navigation is a cornerstone of embodied AI, enabling agents to interpret language instructions and navigate complex environments. However, expert-provided instructions are limited in quantity, while synthesized annotations often lack quality, making them insufficient for large-scale research. To address this, we propose NavComposer, a novel framework for automatically generating high-quality navigation instructions. NavComposer explicitly decomposes semantic entities such as actions, scenes, and objects, and recomposes them into natural language instructions. Its modular architecture allows flexible integration of state-of-the-art techniques, while the explicit use of semantic entities enhances both the richness and accuracy of instructions. Moreover, it operates in a data-agnostic manner, supporting adaptation to diverse navigation trajectories without domain-specific training. Complementing NavComposer, we introduce NavInstrCritic, a comprehensive annotation-free evaluation system that assesses navigation instructions on three dimensions: contrastive matching, semantic consistency, and linguistic diversity. NavInstrCritic provides a holistic evaluation of instruction quality, addressing limitations of traditional metrics that rely heavily on expert annotations. By decoupling instruction generation and evaluation from specific navigation agents, our method enables more scalable and generalizable research. Extensive experiments provide direct and practical evidence for the effectiveness of our method.

Related papers

• LangNavBench: Evaluation of Natural Language Understanding in Semantic Navigation [18.951580080771432]
  LangNav is an open-set dataset specifically created to test an agent's ability to locate objects described at different levels of detail.<n>LangNavBench allows us to systematically compare models on their handling of attributes, spatial and relational cues, and category hierarchies.<n>MLFM is a method that builds a queryable multi-layered semantic map.
  arXiv  Detail & Related papers  (2025-07-09T21:46:43Z)
• NAVCON: A Cognitively Inspired and Linguistically Grounded Corpus for Vision and Language Navigation [66.89717229608358]
  NAVCON is a large-scale annotated Vision-Language Navigation (VLN) corpus built on top of two popular datasets (R2R and RxR)
  arXiv  Detail & Related papers  (2024-12-17T15:48:25Z)
• SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts [54.11162991206203]
  This paper consolidates diverse navigation tasks into a unified and generic framework.<n>We propose a novel State-Adaptive Mixture of Experts (SAME) model that effectively enables an agent to infer decisions.
  arXiv  Detail & Related papers  (2024-12-07T06:12:53Z)
• NavHint: Vision and Language Navigation Agent with a Hint Generator [31.322331792911598]
  We provide indirect supervision to the navigation agent through a hint generator that provides detailed visual descriptions. The hint generator assists the navigation agent in developing a global understanding of the visual environment. We evaluate our method on the R2R and R4R datasets and achieve state-of-the-art on several metrics.
  arXiv  Detail & Related papers  (2024-02-04T16:23:16Z)
• $A^2$Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models [89.64729024399634]
  We study the task of zero-shot vision-and-language navigation (ZS-VLN), a practical yet challenging problem in which an agent learns to navigate following a path described by language instructions. Normally, the instructions have complex grammatical structures and often contain various action descriptions. How to correctly understand and execute these action demands is a critical problem, and the absence of annotated data makes it even more challenging.
  arXiv  Detail & Related papers  (2023-08-15T19:01:19Z)
• Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigation [172.15808300686584]
  We describe an approach that learns the two tasks simultaneously and exploits their intrinsic correlations to boost the training of each. Our approach improves the performance of various follower models and produces accurate navigation instructions.
  arXiv  Detail & Related papers  (2022-03-30T18:15:26Z)
• Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation [145.84123197129298]
  Language instruction plays an essential role in the natural language grounded navigation tasks. We exploit to train a more robust navigator which is capable of dynamically extracting crucial factors from the long instruction. Specifically, we propose a Dynamic Reinforced Instruction Attacker (DR-Attacker), which learns to mislead the navigator to move to the wrong target.
  arXiv  Detail & Related papers  (2021-07-23T14:11:31Z)
• Diagnosing Vision-and-Language Navigation: What Really Matters [61.72935815656582]
  Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Recent studies witness a slow-down in the performance improvements in both indoor and outdoor VLN tasks. In this work, we conduct a series of diagnostic experiments to unveil agents' focus during navigation.
  arXiv  Detail & Related papers  (2021-03-30T17:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.