Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigation

• URL: http://arxiv.org/abs/2603.26859v1
• Date: Fri, 27 Mar 2026 15:25:30 GMT
• Title: Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigation
• Abstract summary: Vision-and-Language Navigation (VLN) requires an agent to navigate through complex unseen environments.<n>We propose Beyond Textual Knowledge (BTK), a VLN framework that integrates environment-specific textual knowledge with generative image knowledge bases.
• Score: 20.499082957224925
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-and-Language Navigation (VLN) requires an agent to navigate through complex unseen environments based on natural language instructions. However, existing methods often struggle to effectively capture key semantic cues and accurately align them with visual observations. To address this limitation, we propose Beyond Textual Knowledge (BTK), a VLN framework that synergistically integrates environment-specific textual knowledge with generative image knowledge bases. BTK employs Qwen3-4B to extract goal-related phrases and utilizes Flux-Schnell to construct two large-scale image knowledge bases: R2R-GP and REVERIE-GP. Additionally, we leverage BLIP-2 to construct a large-scale textual knowledge base derived from panoramic views, providing environment-specific semantic cues. These multimodal knowledge bases are effectively integrated via the Goal-Aware Augmentor and Knowledge Augmentor, significantly enhancing semantic grounding and cross-modal alignment. Extensive experiments on the R2R dataset with 7,189 trajectories and the REVERIE dataset with 21,702 instructions demonstrate that BTK significantly outperforms existing baselines. On the test unseen splits of R2R and REVERIE, SR increased by 5% and 2.07% respectively, and SPL increased by 4% and 3.69% respectively. The source code is available at https://github.com/yds3/IPM-BTK/.

Related papers

• NAIMA: Semantics Aware RGB Guided Depth Super-Resolution [50.15623093332659]
  Guided depth super-resolution (GDSR) is a multi-modal approach for depth map super-resolution that relies on a low-resolution depth map and a high-resolution RGB image to restore finer structural details.<n>We propose a solution that introduces global contextual semantic priors, generated from pretrained vision transformer token embeddings.
  arXiv  Detail & Related papers  (2026-04-06T04:14:59Z)
• MMLGNet: Cross-Modal Alignment of Remote Sensing Data using CLIP [21.89022894877594]
  We propose a novel framework, Multimodal Language-Guided Network (MMLGNet), to align heterogeneous remote sensing modalities with natural language semantics.<n>Inspired by CLIP's training paradigm, our approach bridges the gap between high-dimensional remote sensing data and language-guided interpretation.
  arXiv  Detail & Related papers  (2026-01-13T10:44:37Z)
• Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering [75.12322966980003]
  Large Language Models (LLMs) have shown strong inductive reasoning ability across various domains.<n>Most existing RAG pipelines rely on unstructured text, limiting interpretability and structured reasoning.<n>Recent studies have explored integrating knowledge graphs with LLMs for knowledge graph question answering.<n>We propose RAPL, a novel framework for efficient and effective graph retrieval in KGQA.
  arXiv  Detail & Related papers  (2025-06-11T12:03:52Z)
• SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning [48.64616652818972]
  We propose a novel framework that transforms raw depth data into structured, interpretable textual rationales.<n>These textual rationales serve as meaningful intermediate representations to significantly enhance spatial reasoning capabilities.<n>We introduce a new dataset named SSR-CoT, a million-scale visual-language reasoning dataset enriched with intermediate spatial reasoning annotations.
  arXiv  Detail & Related papers  (2025-05-18T14:40:16Z)
• Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speech [3.391256280235937]
  Visual Text-to-Speech (VTTS) aims to take the environmental image as the prompt to synthesize reverberant speech for the spoken content.<n>We propose a novel multi-source spatial knowledge understanding scheme for immersive VTTS, termed MS2KU-VTTS.
  arXiv  Detail & Related papers  (2024-10-18T00:46:18Z)
• Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generation [14.448198170932226]
  Think-on-Graph 2.0 (ToG-2) is a hybrid RAG framework that iteratively retrieves information from both unstructured and structured knowledge sources.<n>ToG-2 alternates between graph retrieval and context retrieval to search for in-depth clues relevant to the question.<n>It achieves overall state-of-the-art (SOTA) performance on 6 out of 7 knowledge-intensive datasets with GPT-3.5.
  arXiv  Detail & Related papers  (2024-07-15T15:20:40Z)
• LuoJiaHOG: A Hierarchy Oriented Geo-aware Image Caption Dataset for Remote Sensing Image-Text Retrival [8.656768875730904]
  We introduce an image caption dataset LuojiaHOG, which is geospatial-aware, label-extension-friendly and comprehensive-captioned. LuojiaHOG involves the hierarchical spatial sampling, classification system to Open Geospatial Consortium (OGC) standards, and detailed caption generation. We also propose a CLIP-based Image Semantic Enhancement Network (CISEN) to promote sophisticated ITR.
  arXiv  Detail & Related papers  (2024-03-16T10:46:14Z)
• BEV$^2$PR: BEV-Enhanced Visual Place Recognition with Structural Cues [44.96177875644304]
  We propose a new image-based visual place recognition (VPR) framework by exploiting the structural cues in bird's-eye view (BEV) from a single camera. The BEV$2$PR framework generates a composite descriptor with both visual cues and spatial awareness based on a single camera.
  arXiv  Detail & Related papers  (2024-03-11T10:46:43Z)
• Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model [63.461030694700014]
  We propose a novel dual knowledge-enhanced generative pretrained language model for multimodal task-oriented dialog systems (DKMD) The proposed DKMD consists of three key components: dual knowledge selection, dual knowledge-enhanced context learning, and knowledge-enhanced response generation. Experiments on a public dataset verify the superiority of the proposed DKMD over state-of-the-art competitors.
  arXiv  Detail & Related papers  (2022-07-16T13:02:54Z)
• Bifurcated backbone strategy for RGB-D salient object detection [168.19708737906618]
  We leverage the inherent multi-modal and multi-level nature of RGB-D salient object detection to devise a novel cascaded refinement network. Our architecture, named Bifurcated Backbone Strategy Network (BBS-Net), is simple, efficient, and backbone-independent.
  arXiv  Detail & Related papers  (2020-07-06T13:01:30Z)
• Multi-View Learning for Vision-and-Language Navigation [163.20410080001324]
  Learn from EveryOne (LEO) is a training paradigm for learning to navigate in a visual environment. By sharing parameters across instructions, our approach learns more effectively from limited training data. On the recent Room-to-Room (R2R) benchmark dataset, LEO achieves 16% improvement (absolute) over a greedy agent.
  arXiv  Detail & Related papers  (2020-03-02T13:07:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.