VENTURA: Adapting Image Diffusion Models for Unified Task Conditioned Navigation

• URL: http://arxiv.org/abs/2510.01388v1
• Date: Wed, 01 Oct 2025 19:21:28 GMT
• Title: VENTURA: Adapting Image Diffusion Models for Unified Task Conditioned Navigation
• Authors: Arthur Zhang, Xiangyun Meng, Luca Calliari, Dong-Ki Kim, Shayegan Omidshafiei, Joydeep Biswas, Ali Agha, Amirreza Shaban,
• Abstract summary: VENTURA is a vision-supervised navigation system that finetunes internet-pretrained image diffusion models for path planning.<n>A lightweight behavior-cloning policy grounds these visual plans into executable trajectories, yielding an interface that follows natural language instructions.<n>In extensive real-world evaluations, VENTURA outperforms state-of-the-art foundation model baselines on object reaching, obstacle avoidance, and terrain preference tasks.
• Score: 15.811034169990423
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Robots must adapt to diverse human instructions and operate safely in unstructured, open-world environments. Recent Vision-Language models (VLMs) offer strong priors for grounding language and perception, but remain difficult to steer for navigation due to differences in action spaces and pretraining objectives that hamper transferability to robotics tasks. Towards addressing this, we introduce VENTURA, a vision-language navigation system that finetunes internet-pretrained image diffusion models for path planning. Instead of directly predicting low-level actions, VENTURA generates a path mask (i.e. a visual plan) in image space that captures fine-grained, context-aware navigation behaviors. A lightweight behavior-cloning policy grounds these visual plans into executable trajectories, yielding an interface that follows natural language instructions to generate diverse robot behaviors. To scale training, we supervise on path masks derived from self-supervised tracking models paired with VLM-augmented captions, avoiding manual pixel-level annotation or highly engineered data collection setups. In extensive real-world evaluations, VENTURA outperforms state-of-the-art foundation model baselines on object reaching, obstacle avoidance, and terrain preference tasks, improving success rates by 33% and reducing collisions by 54% across both seen and unseen scenarios. Notably, we find that VENTURA generalizes to unseen combinations of distinct tasks, revealing emergent compositional capabilities. Videos, code, and additional materials: https://venturapath.github.io

Related papers

• OpenFrontier: General Navigation with Visual-Language Grounded Frontiers [54.661157616245966]
  Open-world navigation requires robots to make decisions in complex everyday environments.<n>Recent advances in vision--language navigation (VLN) and vision--language--action (VLA) models enable end-to-end policies conditioned on natural language.<n>We propose OpenFrontier, a training-free navigation framework that seamlessly integrates diverse vision--language prior models.
  arXiv  Detail & Related papers  (2026-03-05T17:02:22Z)
• VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigation [16.279434375658457]
  VAMOS is a hierarchical VLA that decouples semantic planning from embodiment grounding.<n>We show VAMOS achieves higher success rates in both indoor and complex outdoor navigation.<n>This model significantly enhances single-robot reliability, achieving 3X higher success rates by rejecting physically infeasible plans.
  arXiv  Detail & Related papers  (2025-10-23T17:59:45Z)
• Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environments [10.953629652228024]
  Vision-and-Language Navigation (VLN) agents associate time-sequenced visual observations with corresponding instructions to make decisions.<n>In this paper, we address the mismatch between human-centric instructions and quadruped robots with a low-height field of view.<n>We propose a Ground-level Viewpoint Navigation (GVNav) approach to mitigate this issue.
  arXiv  Detail & Related papers  (2025-02-26T10:30:40Z)
• NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistants [24.689242976554482]
  Navigating unfamiliar environments presents significant challenges for household robots.<n>Existing reinforcement learning methods cannot be directly transferred to new environments.<n>We try to transfer the logical knowledge and the generalization ability of pre-trained foundation models to zero-shot navigation.
  arXiv  Detail & Related papers  (2025-02-19T17:27:47Z)
• Interactive Navigation in Environments with Traversable Obstacles Using Large Language and Vision-Language Models [14.871309526022516]
  This paper proposes an interactive navigation framework by using large language and vision-language models. We create an action-aware costmap to perform effective path planning without fine-tuning. All experimental results demonstrated the proposed framework's effectiveness and adaptability to diverse environments.
  arXiv  Detail & Related papers  (2023-10-13T05:59:03Z)
• Improving Vision-and-Language Navigation by Generating Future-View Image Semantics [96.8435716885159]
  Vision-and-Language Navigation (VLN) is the task that requires an agent to navigate through the environment based on natural language instructions. We propose three proxy tasks during the agent's in-domain pre-training: Masked Panorama Modeling (MPM), Masked Trajectory Modeling (MTM), and Action Prediction with Image Generation (APIG) We then fine-tune the agent on the VLN task with an auxiliary loss that minimizes the difference between the view semantics generated by the agent and the ground truth view semantics of the next step.
  arXiv  Detail & Related papers  (2023-04-11T00:36:02Z)
• A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning [70.14372215250535]
  Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments. Given the scarcity of human instruction data and limited diversity in the training environments, these agents still struggle with complex language grounding and spatial language understanding. We take 500+ indoor environments captured in densely-sampled 360 degree panoramas, construct navigation trajectories through these panoramas, and generate a visually-grounded instruction for each trajectory. The resulting dataset of 4.2M instruction-trajectory pairs is two orders of magnitude larger than existing human-annotated datasets.
  arXiv  Detail & Related papers  (2022-10-06T17:59:08Z)
• LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action [76.71101507291473]
  We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories. We show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data.
  arXiv  Detail & Related papers  (2022-07-10T10:41:50Z)
• Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships [52.72020203771489]
  We investigate target-driven visual navigation using deep reinforcement learning (DRL) in 3D indoor scenes. Our proposed method combines visual features and 3D spatial representations to learn navigation policy. Our experiments, performed in the AI2-THOR, show that our model outperforms the baselines in both SR and SPL metrics.
  arXiv  Detail & Related papers  (2020-04-29T08:46:38Z)
• Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling [65.99956848461915]
  Vision-and-Language Navigation (VLN) is a task where agents must decide how to move through a 3D environment to reach a goal.<n>One of the problems of the VLN task is data scarcity since it is difficult to collect enough navigation paths with human-annotated instructions for interactive environments.<n>We propose an adversarial-driven counterfactual reasoning model that can consider effective conditions instead of low-quality augmented data.
  arXiv  Detail & Related papers  (2019-11-17T18:02:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.