CoINS: Counterfactual Interactive Navigation via Skill-Aware VLM

• URL: http://arxiv.org/abs/2601.03956v1
• Date: Wed, 07 Jan 2026 14:10:46 GMT
• Title: CoINS: Counterfactual Interactive Navigation via Skill-Aware VLM
• Abstract summary: We propose Counterfactual Interactive Navigation via Skill-aware VLM (CoINS)<n>We fine-tune a VLM, named InterNav-VLM, which incorporates skill affordance and concrete constraint parameters into the input context.<n>To execute the generated high-level plans, we develop a comprehensive skill library through reinforcement learning.
• Score: 13.352875026471445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent Vision-Language Models (VLMs) have demonstrated significant potential in robotic planning. However, they typically function as semantic reasoners, lacking an intrinsic understanding of the specific robot's physical capabilities. This limitation is particularly critical in interactive navigation, where robots must actively modify cluttered environments to create traversable paths. Existing VLM-based navigators are predominantly confined to passive obstacle avoidance, failing to reason about when and how to interact with objects to clear blocked paths. To bridge this gap, we propose Counterfactual Interactive Navigation via Skill-aware VLM (CoINS), a hierarchical framework that integrates skill-aware reasoning and robust low-level execution. Specifically, we fine-tune a VLM, named InterNav-VLM, which incorporates skill affordance and concrete constraint parameters into the input context and grounds them into a metric-scale environmental representation. By internalizing the logic of counterfactual reasoning through fine-tuning on the proposed InterNav dataset, the model learns to implicitly evaluate the causal effects of object removal on navigation connectivity, thereby determining interaction necessity and target selection. To execute the generated high-level plans, we develop a comprehensive skill library through reinforcement learning, specifically introducing traversability-oriented strategies to manipulate diverse objects for path clearance. A systematic benchmark in Isaac Sim is proposed to evaluate both the reasoning and execution aspects of interactive navigation. Extensive simulations and real-world experiments demonstrate that CoINS significantly outperforms representative baselines, achieving a 17\% higher overall success rate and over 80\% improvement in complex long-horizon scenarios compared to the best-performing baseline

Related papers

• Vision-Language Procedural Reasoning for Context-Aware Reward Modeling of Robotic Endovascular Guidewire Navigation [6.022893284275472]
  This paper proposes a vision-language procedural reasoning (VL-PR) framework for autonomous guidewire navigation.<n>The framework integrates a multimodal large language model (MLLM) as a procedural reasoning module that interprets real-time visual observations to infer high-level navigation contexts.<n> Experiments on a physical robotic platform across diverse vascular scenarios demonstrate enhanced task reliability and streamlined navigational efficiency.
  arXiv  Detail & Related papers  (2026-06-29T07:35:20Z)
• Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control [27.669648439283936]
  Social robot navigation (SRN) requires understanding human intentions, social norms, and contextual cues to generate socially compliant behaviors.<n>Recent advances in Vision-Language Models (VLMs) have opened new opportunities for SRN.<n>Survey aims to provide a foundation for building reliable, socially compliant, and deployable VLM-enabled navigation systems.
  arXiv  Detail & Related papers  (2026-06-27T06:32:47Z)
• AgentVLN: Towards Agentic Vision-and-Language Navigation [78.739525400071]
  Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments.<n>In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms.
  arXiv  Detail & Related papers  (2026-03-18T12:43:47Z)
• TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigation [70.23578202012048]
  Vision-Language Navigation (VLN) presents a unique challenge for Large Vision-Language Models (VLMs) due to their inherent architectural mismatch.<n>We propose TagaVLM (Topology-Aware Global Action reasoning), an end-to-end framework that explicitly injects topological structures into the VLM backbone.<n>To enhance topological node information, an Interleaved Navigation Prompt strengthens node-level visual-text alignment.<n>With the embedded topological graph, the model is capable of global action reasoning, allowing for robust path correction.
  arXiv  Detail & Related papers  (2026-03-03T13:28:07Z)
• AutoFly: Vision-Language-Action Model for UAV Autonomous Navigation in the Wild [62.47761809929869]
  Vision-language navigation (VLN) requires intelligent agents to navigate environments by interpreting linguistic instructions alongside visual observations.<n>Current VLN research for unmanned aerial vehicles (UAVs) relies on detailed, pre-specified instructions to guide the UAV along predetermined routes.<n>We propose AutoFly, an end-to-end Vision-Language-Action model for autonomous UAV navigation.
  arXiv  Detail & Related papers  (2026-02-10T11:08:07Z)
• VL-LN Bench: Towards Long-horizon Goal-oriented Navigation with Active Dialogs [51.18508300818979]
  Vision Language-Language Navigation (VL-LN) benchmark provides a large-scale, automatically generated dataset for training and assessing dialog-enabled navigation models.<n>VL-LN comprises over 41k long-horizon dialog-augmented trajectories for training and an automatic evaluation protocol with an oracle capable of responding to agent queries.
  arXiv  Detail & Related papers  (2025-12-26T19:00:12Z)
• Think, Remember, Navigate: Zero-Shot Object-Goal Navigation with VLM-Powered Reasoning [8.316876835702525]
  Vision-Language Models (VLMs) are set to transform robotic navigation.<n>Our framework outsources high-level planning to a VLM, which leverages its contextual understanding to guide a frontier-based exploration agent.<n>When tested on challenging benchmarks like HM3D, Gibson, and MP3D, this method produces exceptionally direct and logical trajectories.
  arXiv  Detail & Related papers  (2025-11-12T03:38:50Z)
• General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Acting [5.291702442384798]
  Agentic Robotic Navigation Architecture (ARNA) is a general-purpose framework that equips an LVLM-based agent with a library of perception, reasoning, and navigation tools.<n>At runtime, the agent autonomously defines and executes task-specific navigation tools drawn from modern robotic stacks.<n>ARNA outperforms state-of-the-art EQA-specific approaches.
  arXiv  Detail & Related papers  (2025-06-20T20:06:14Z)
• History-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigation [5.343932820859596]
  This paper introduces a novel zero-shot ObjectNav framework that pioneers the use of dynamic, history-aware prompting.<n>Our core innovation lies in providing the VLM with action history context, enabling it to generate semantic guidance scores for navigation actions.<n>We also introduce a VLM-assisted waypoint generation mechanism for refining the final approach to detected objects.
  arXiv  Detail & Related papers  (2025-06-19T21:50:16Z)
• Navigating Motion Agents in Dynamic and Cluttered Environments through LLM Reasoning [69.5875073447454]
  This paper advances motion agents empowered by large language models (LLMs) toward autonomous navigation in dynamic and cluttered environments.<n>Our training-free framework supports multi-agent coordination, closed-loop replanning, and dynamic obstacle avoidance without retraining or fine-tuning.
  arXiv  Detail & Related papers  (2025-03-10T13:39:09Z)
• Cognitive Planning for Object Goal Navigation using Generative AI Models [0.979851640406258]
  We present a novel framework for solving the object goal navigation problem that generates efficient exploration strategies. Our approach enables a robot to navigate unfamiliar environments by leveraging Large Language Models (LLMs) and Large Vision-Language Models (LVLMs)
  arXiv  Detail & Related papers  (2024-03-30T10:54:59Z)
• NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning [97.88246428240872]
  Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.<n>Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability.<n>This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision.
  arXiv  Detail & Related papers  (2024-03-12T07:27:02Z)
• ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments [56.194988818341976]
  Vision-language navigation is a task that requires an agent to follow instructions to navigate in environments. We propose ETPNav, which focuses on two critical skills: 1) the capability to abstract environments and generate long-range navigation plans, and 2) the ability of obstacle-avoiding control in continuous environments. ETPNav yields more than 10% and 20% improvements over prior state-of-the-art on R2R-CE and RxR-CE datasets.
  arXiv  Detail & Related papers  (2023-04-06T13:07:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.