LaF-GRPO: In-Situ Navigation Instruction Generation for the Visually Impaired via GRPO with LLM-as-Follower Reward

• URL: http://arxiv.org/abs/2506.04070v1
• Date: Wed, 04 Jun 2025 15:34:33 GMT
• Title: LaF-GRPO: In-Situ Navigation Instruction Generation for the Visually Impaired via GRPO with LLM-as-Follower Reward
• Abstract summary: This study focuses on producing precise, in-situ, step-by-step navigation instructions that are practically usable by visually impaired (VI) users.<n>LLM-as-Follower GRPO generates rewards guiding the Vision-Language Model (VLM) post-training.<n>Experiments on NIG4VI show the effectiveness of LaF-GRPO by quantitative metrics.
• Score: 10.432056588923224
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Navigation instruction generation for visually impaired (VI) individuals (NIG-VI) is critical yet relatively underexplored. This study, hence, focuses on producing precise, in-situ, step-by-step navigation instructions that are practically usable by VI users. Concretely, we propose LaF-GRPO (LLM-as-Follower GRPO), where an LLM simulates VI user responses to generate rewards guiding the Vision-Language Model (VLM) post-training. This enhances instruction usability while reducing costly real-world data needs. To facilitate training and testing, we introduce NIG4VI, a 27k-sample open-sourced benchmark. It provides diverse navigation scenarios with accurate spatial coordinates, supporting detailed, open-ended in-situ instruction generation. Experiments on NIG4VI show the effectiveness of LaF-GRPO by quantitative metrics (e.g., Zero-(LaF-GRPO) boosts BLEU +14\%; SFT+(LaF-GRPO) METEOR 0.542 vs. GPT-4o's 0.323) and yields more intuitive, safer instructions. Code and benchmark are available at \href{https://github.com/YiyiyiZhao/NIG4VI}{https://github.com/YiyiyiZhao/NIG4VI}.

Related papers

• VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments [14.629408091049982]
  We propose VTInstructor, the first VLN instruction generation framework for continuous environments.<n> EDTC condenses long RGB trajectories into navigation-criticals, VTP overlays path, turn, and goal cues onto these anchors, and VTMod injects the resulting trajectory signals into the visual encoder.<n>On the challenging R2R-CE and RxR-CE Val Unseen benchmarks, VTInstructor sets a new state of the art across all standard NLG metrics.
  arXiv  Detail & Related papers  (2026-08-15T15:31:27Z)
• pFedNavi: Structure-Aware Personalized Federated Vision-Language Navigation for Embodied AI [27.929546538010516]
  Vision-Language Navigation VLN requires large-scale trajectory instruction data from private indoor environments.<n> Federated Learning FL mitigates this by keeping data on-device, but vanilla FL struggles under VLNs' extreme cross-client heterogeneity.<n>This paper proposes pFedNavi, a structure-aware and dynamically adaptive personalized federated learning framework tailored for VLN.
  arXiv  Detail & Related papers  (2026-02-16T02:18:09Z)
• Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels [30.722073025794025]
  We address five critical real-world challenges in text-instruction-based grounding.<n>Our approach generates high-quality instruction-response pairs linked to existing pixel-level annotations.<n>Experiment results show that models trained on Ground-V exhibit substantial improvements across diverse grounding tasks.
  arXiv  Detail & Related papers  (2025-05-20T00:37:19Z)
• SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models [15.50826328938879]
  We introduce SURDS, a benchmark designed to evaluate the spatial reasoning capabilities of vision language models (VLMs)<n>Built on the nuScenes dataset, SURDS comprises 41,080 vision-question-answer training instances and 9,250 evaluation samples.<n>We propose a reinforcement learning-based alignment scheme leveraging spatially grounded reward signals.
  arXiv  Detail & Related papers  (2024-11-20T08:14:01Z)
• DivScene: Benchmarking LVLMs for Object Navigation with Diverse Scenes and Objects [84.73092715537364]
  In this paper, we study a new task of navigating to diverse target objects in a large number of scene types. We build an end-to-end embodied agent, NatVLM, by fine-tuning a Large Vision Language Model (LVLM) through imitation learning. Our agent achieves a success rate that surpasses GPT-4o by over 20%.
  arXiv  Detail & Related papers  (2024-10-03T17:49:28Z)
• NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning [97.88246428240872]
  Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.<n>Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability.<n>This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision.
  arXiv  Detail & Related papers  (2024-03-12T07:27:02Z)
• GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition? [82.40761196684524]
  This paper centers on the evaluation of GPT-4's linguistic and visual capabilities in zero-shot visual recognition tasks. We conduct extensive experiments to evaluate GPT-4's performance across images, videos, and point clouds. Our findings show that GPT-4, enhanced with rich linguistic descriptions, significantly improves zero-shot recognition.
  arXiv  Detail & Related papers  (2023-11-27T11:29:10Z)
• To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning [82.34463739289892]
  We introduce a fine-grained visual instruction dataset, LVIS-Instruct4V, which contains 220K visually aligned and context-aware instructions. By simply replacing the LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA on most challenging LMM benchmarks.
  arXiv  Detail & Related papers  (2023-11-13T18:59:31Z)
• GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond [29.778018058541676]
  GPT-Fathom is an open-source and reproducible evaluation suite for large language models (LLMs) built on top of OpenAI Evals. We evaluate 10+ leading LLMs as well as OpenAI's legacy models on 20+ curated benchmarks across 7 capability categories, all aligned under settings.
  arXiv  Detail & Related papers  (2023-09-28T16:43:35Z)
• Using LLMs to Facilitate Formal Verification of RTL [7.45362896000302]
  We show that GPT4 can generate correct SVA even for flawed RTL, without mirroring design errors. We extend the open-source AutoSVA framework by integrating our improved GPT4-based flow to generate safety properties.
  arXiv  Detail & Related papers  (2023-09-18T02:37:43Z)
• Towards Building the Federated GPT: Federated Instruction Tuning [66.7900343035733]
  This paper introduces Federated Instruction Tuning (FedIT) as the learning framework for the instruction tuning of large language models (LLMs) We demonstrate that by exploiting the heterogeneous and diverse sets of instructions on the client's end with FedIT, we improved the performance of LLMs compared to centralized training with only limited local instructions.
  arXiv  Detail & Related papers  (2023-05-09T17:42:34Z)
• Visual Instruction Tuning [79.70923292053097]
  We present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%.
  arXiv  Detail & Related papers  (2023-04-17T17:59:25Z)
• ULN: Towards Underspecified Vision-and-Language Navigation [77.81257404252132]
  Underspecified vision-and-Language Navigation (ULN) is a new setting for vision-and-Language Navigation (VLN) We propose a VLN framework that consists of a classification module, a navigation agent, and an Exploitation-to-Exploration (E2E) module. Our framework is more robust and outperforms the baselines on ULN by 10% relative success rate across all levels.
  arXiv  Detail & Related papers  (2022-10-18T17:45:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.