Teaching Vision-Language-Action Models What to See and Where to Look

• URL: http://arxiv.org/abs/2607.01658v1
• Date: Thu, 02 Jul 2026 03:34:32 GMT
• Title: Teaching Vision-Language-Action Models What to See and Where to Look
• Abstract summary: Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving.<n>We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look.
• Score: 40.70324197562915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Related papers

• Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving [68.31544904268218]
  We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving.<n>Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework.
  arXiv  Detail & Related papers  (2026-08-31T17:59:54Z)
• DriveStack-VLA: Render-Teacher Alignment for BEV-Based DeepStack Vision-Language-Action Model [22.06522773748522]
  We present DriveStack-VLA, a framework built upon a large VLM backbone.<n>We develop dual visual modeling components to strengthen the spatial grounding of VLA driving.<n>DriveStack-VLA achieves 91.6 PDMS on NAVSIMv1, 91.0S on NAVSIMv2, and a driving score of 79.49 with a success rate of 56.36% on the closed-loop Bench2Drive.
  arXiv  Detail & Related papers  (2026-06-23T01:40:54Z)
• AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation [78.80536515102305]
  Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment.<n>We propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism.<n>Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training.
  arXiv  Detail & Related papers  (2026-05-21T17:58:26Z)
• LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model [16.87741001074065]
  Vision-Language-Action (VLA) models have emerged as a promising framework for end-to-end autonomous driving.<n>Recent attempts to incorporate dense visual supervision via world modeling often overemphasize pixel-level image reconstruction.<n>We propose LVDrive, a Latent Visual representation enhanced VLA framework for autonomous driving.
  arXiv  Detail & Related papers  (2026-05-21T07:31:49Z)
• AutoFly: Vision-Language-Action Model for UAV Autonomous Navigation in the Wild [62.47761809929869]
  Vision-language navigation (VLN) requires intelligent agents to navigate environments by interpreting linguistic instructions alongside visual observations.<n>Current VLN research for unmanned aerial vehicles (UAVs) relies on detailed, pre-specified instructions to guide the UAV along predetermined routes.<n>We propose AutoFly, an end-to-end Vision-Language-Action model for autonomous UAV navigation.
  arXiv  Detail & Related papers  (2026-02-10T11:08:07Z)
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future [125.92052530850425]
  Vision-Language-Action (VLA) frameworks integrate perception with language-grounded decision making.<n>VLA frameworks offer a pathway toward more interpretable, generalizable, and human-aligned driving policies.<n>This work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.
  arXiv  Detail & Related papers  (2025-12-18T16:57:44Z)
• OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model [24.90085777003393]
  We present OpenDriveVLA, a Vision-Language Action (VLA) model designed for end-to-end autonomous driving.<n>OpenDriveVLA builds upon open-source pre-trained large Vision-Language Models (VLMs) to generate reliable driving actions.
  arXiv  Detail & Related papers  (2025-03-30T14:45:54Z)
• OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction [95.6266030753644]
  Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions.<n>Existing approaches require fine-tuning pre-trained vision-language models (VLMs) as visual and language features are independently fed into downstream policies.<n>We propose OTTER, a novel VLA architecture that leverages existing alignments through explicit, text-aware visual feature extraction.
  arXiv  Detail & Related papers  (2025-03-05T18:44:48Z)
• Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving [20.33096710167997]
  generative planning with 3D-vision language pre-training model named GPVL is proposed for end-to-end autonomous driving.<n>Cross-modal language model is introduced to generate holistic driving decisions and fine-grained trajectories.<n>It is believed that the effective, robust and efficient performance of GPVL is crucial for the practical application of future autonomous driving systems.
  arXiv  Detail & Related papers  (2025-01-15T15:20:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.