SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios

• URL: http://arxiv.org/abs/2602.08440v2
• Date: Fri, 13 Feb 2026 08:14:33 GMT
• Title: SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios
• Abstract summary: A fundamental challenge in autonomous driving is the integration of high-level, semantic reasoning for long-tail events with low-level, reactive control for robust driving.<n>We propose SteerVLA, which produces fine-grained language instructions that steer a vision-language-action driving policy.<n>We evaluate SteerVLA on a challenging closed-loop benchmark, where it outperforms state-of-the-art methods by 4.77 points in overall driving score and by 8.04 points on a long-tail subset.
• Score: 104.10555123175055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A fundamental challenge in autonomous driving is the integration of high-level, semantic reasoning for long-tail events with low-level, reactive control for robust driving. While large vision-language models (VLMs) trained on web-scale data offer powerful common-sense reasoning, they lack the grounded experience necessary for safe vehicle control. We posit that an effective autonomous agent should leverage the world knowledge of VLMs to guide a steerable driving policy toward robust control in driving scenarios. To this end, we propose SteerVLA, which leverages the reasoning capabilities of VLMs to produce fine-grained language instructions that steer a vision-language-action (VLA) driving policy. Key to our method is this rich language interface between the high-level VLM and low-level VLA, which allows the high-level policy to more effectively ground its reasoning in the control outputs of the low-level policy. To provide fine-grained language supervision aligned with vehicle control, we leverage a VLM to augment existing driving data with detailed language annotations, which we find to be essential for effective reasoning and steerability. We evaluate SteerVLA on a challenging closed-loop benchmark, where it outperforms state-of-the-art methods by 4.77 points in overall driving score and by 8.04 points on a long-tail subset. The project website is available at: https://steervla.github.io/.

Related papers

• Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control [46.169163284648384]
  We introduce Steerable Policies: VLAs trained on rich synthetic commands at various levels of abstraction, like subtasks, motions, and grounded pixel coordinates.<n>We demonstrate this benefit by controlling our Steerable Policies with both a learned high-level embodied reasoner and an off-the-shelf VLM prompted to reason over command abstractions via in-context learning.
  arXiv  Detail & Related papers  (2026-02-13T18:57:56Z)
• dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning [69.36145467833498]
  We introduce dVLM-AD, a diffusion-based vision-language model that unifies perception, structured reasoning, and low-level planning for end-to-end driving.<n> evaluated on nuScenes and WOD-E2E, dVLM-AD yields more consistent reasoning-action pairs and achieves planning performance comparable to existing driving VLM/VLA systems.
  arXiv  Detail & Related papers  (2025-12-04T05:05:41Z)
• Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving [46.99350914451702]
  Reasoning-VLA achieves state-of-the-art performance, superior generalization capability, and the excellent inference speed reported to date.<n>We consolidate eight publicly available autonomous driving datasets into a standardized, Chain-of-language reasoning-based, and easy-to-use data format for model training.
  arXiv  Detail & Related papers  (2025-11-25T04:40:11Z)
• A Survey on Vision-Language-Action Models for Autonomous Driving [26.407082158880204]
  Vision-Language-Action (VLA) paradigms integrate visual perception, natural language understanding, and control within a single policy.<n>Researchers in autonomous driving are actively adapting these methods to the vehicle domain.<n>This survey offers the first comprehensive overview of VLA for Autonomous Driving.
  arXiv  Detail & Related papers  (2025-06-30T16:50:02Z)
• Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving [19.48508500497233]
  We present Poutine, a method that uses an off-the-shelf vision-language model (VLM) to achieve robust end-to-end autonomous driving.<n>To learn strong base driving capabilities, we first train Poutine-Base using self-supervised next-token prediction over vision, language, and trajectory (VLT) tokens.<n>The final Poutine model achieves an RFS of 7.99 on the test set, placing 1st in the 2025 Vision-Based End-to-End Driving Challenge.
  arXiv  Detail & Related papers  (2025-06-12T19:14:00Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving [0.4371652524921044]
  CoVLA (Comprehensive Vision-Language-Action) dataset comprises real-world driving videos spanning more than 80 hours.<n>This dataset establishes a framework for robust, interpretable, and data-driven autonomous driving systems.
  arXiv  Detail & Related papers  (2024-08-19T09:53:49Z)
• Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases [102.05741859030951]
  We propose CODA-LM, the first benchmark for the automatic evaluation of LVLMs for self-driving corner cases.<n>We show that using the text-only large language models as judges reveals even better alignment with human preferences than the LVLM judges.<n>Our CODA-VLM performs comparably with GPT-4V, even surpassing GPT-4V by +21.42% on the regional perception task.
  arXiv  Detail & Related papers  (2024-04-16T14:20:55Z)
• VLP: Vision Language Planning for Autonomous Driving [52.640371249017335]
  This paper presents a novel Vision-Language-Planning framework that exploits language models to bridge the gap between linguistic understanding and autonomous driving. It achieves state-of-the-art end-to-end planning performance on the NuScenes dataset by achieving 35.9% and 60.5% reduction in terms of average L2 error and collision rates, respectively.
  arXiv  Detail & Related papers  (2024-01-10T23:00:40Z)
• DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving [69.82743399946371]
  DriveMLM is a framework that can perform close-loop autonomous driving in realistic simulators. We employ a multi-modal LLM (MLLM) to model the behavior planning module of a module AD system. This model can plug-and-play in existing AD systems such as Apollo for close-loop driving.
  arXiv  Detail & Related papers  (2023-12-14T18:59:05Z)
• DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model [84.29836263441136]
  This study introduces DriveGPT4, a novel interpretable end-to-end autonomous driving system based on multimodal large language models (MLLMs) DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users.
  arXiv  Detail & Related papers  (2023-10-02T17:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.