DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving

• URL: http://arxiv.org/abs/2505.20665v1
• Date: Tue, 27 May 2025 03:21:04 GMT
• Title: DriveRX: A Vision-Language Reasoning Model for Cross-Task Autonomous Driving
• Authors: Muxi Diao, Lele Yang, Hongbo Yin, Zhexu Wang, Yejie Wang, Daxin Tian, Kongming Liang, Zhanyu Ma,
• Abstract summary: We present AutoDriveRL, a unified training framework that formulates autonomous driving as a structured reasoning process over four core tasks.<n>Within this framework, we train DriveRX, a cross-task reasoning VLM designed for real-time decision-making.<n>Our analysis highlights the impact of vision encoder design and reward-guided reasoning compression.
• Score: 22.293019898794963
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous driving requires real-time, robust reasoning across perception, prediction, planning, and behavior. However, conventional end-to-end models fail to generalize in complex scenarios due to the lack of structured reasoning. Recent vision-language models (VLMs) have been applied to driving tasks, but they typically rely on isolated modules and static supervision, limiting their ability to support multi-stage decision-making. We present AutoDriveRL, a unified training framework that formulates autonomous driving as a structured reasoning process over four core tasks. Each task is independently modeled as a vision-language question-answering problem and optimized using task-specific reward models, enabling fine-grained reinforcement signals at different reasoning stages. Within this framework, we train DriveRX, a cross-task reasoning VLM designed for real-time decision-making. DriveRX achieves strong performance on a public benchmark, outperforming GPT-4o in behavior reasoning and demonstrating robustness under complex or corrupted driving conditions. Our analysis further highlights the impact of vision encoder design and reward-guided reasoning compression. We will release the AutoDriveRL framework and the DriveRX model to support future research.

Related papers

• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning [42.409352964719204]
  Vision-Language-Action (VLA) models have shown promise for end-to-end autonomous driving.<n>Current VLA models struggle with physically infeasible action outputs, complex model structures, or unnecessarily long reasoning.<n>We propose AutoVLA, a novel VLA model that unifies reasoning and action generation within a single autoregressive generation model.
  arXiv  Detail & Related papers  (2025-06-16T17:58:50Z)
• Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving [55.96227460521096]
  Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities.<n>We propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios.<n>Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving.
  arXiv  Detail & Related papers  (2025-05-09T20:28:17Z)
• X-Driver: Explainable Autonomous Driving with Vision-Language Models [6.053632514335829]
  End-to-end autonomous driving has advanced significantly, offering benefits such as system simplicity and stronger driving performance.<n>Existing frameworks still suffer from low success rates in closed-loop evaluations, highlighting their limitations in real-world deployment.<n>We introduce X-Driver, a unified multi-modal large language models framework designed for closed-loop autonomous driving.
  arXiv  Detail & Related papers  (2025-05-08T09:52:55Z)
• ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models [9.316712964093506]
  Vision-language models (VLMs) show promise for autonomous driving but often lack transparent reasoning capabilities that are critical for safety.<n>We investigate whether explicitly modeling reasoning during fine-tuning enhances VLM performance on driving decision tasks.
  arXiv  Detail & Related papers  (2025-04-14T23:16:07Z)
• DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding [76.3876070043663]
  We propose DriveLMM-o1, a dataset and benchmark designed to advance step-wise visual reasoning for autonomous driving.<n>Our benchmark features over 18k VQA examples in the training set and more than 4k in the test set, covering diverse questions on perception, prediction, and planning.<n>Our model achieves a +7.49% gain in final answer accuracy, along with a 3.62% improvement in reasoning score over the previous best open-source model.
  arXiv  Detail & Related papers  (2025-03-13T17:59:01Z)
• TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learning [61.33599727106222]
  TeLL-Drive is a hybrid framework that integrates a Teacher LLM to guide an attention-based Student DRL policy.<n>A self-attention mechanism then fuses these strategies with the DRL agent's exploration, accelerating policy convergence and boosting robustness.
  arXiv  Detail & Related papers  (2025-02-03T14:22:03Z)
• VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision [20.43366384946928]
  Vision-language models (VLMs) as teachers to enhance training.<n>VLM-AD achieves significant improvements in planning accuracy and reduced collision rates on the nuScenes dataset.
  arXiv  Detail & Related papers  (2024-12-19T01:53:36Z)
• DRIVE: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Driving [1.4104119587524289]
  Recent advancements in autonomous driving have seen a paradigm shift towards end-to-end learning paradigms. These models often sacrifice interpretability, posing significant challenges to trust, safety, and regulatory compliance. We introduce DRIVE, a comprehensive framework designed to improve the dependability and stability of explanations in end-to-end unsupervised driving models.
  arXiv  Detail & Related papers  (2024-09-16T14:40:47Z)
• DiFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Driving [55.53171248839489]
  We propose an ego-centric fully sparse paradigm, named DiFSD, for end-to-end self-driving.<n>Specifically, DiFSD mainly consists of sparse perception, hierarchical interaction and iterative motion planner.<n>Experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superior planning performance and great efficiency of DiFSD.
  arXiv  Detail & Related papers  (2024-09-15T15:55:24Z)
• DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model [84.29836263441136]
  This study introduces DriveGPT4, a novel interpretable end-to-end autonomous driving system based on multimodal large language models (MLLMs) DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users.
  arXiv  Detail & Related papers  (2023-10-02T17:59:52Z)
• Planning-oriented Autonomous Driving [60.93767791255728]
  We argue that a favorable framework should be devised and optimized in pursuit of the ultimate goal, i.e., planning of the self-driving car. We introduce Unified Autonomous Driving (UniAD), a comprehensive framework that incorporates full-stack driving tasks in one network.
  arXiv  Detail & Related papers  (2022-12-20T10:47:53Z)
• Reason induced visual attention for explainable autonomous driving [2.090380922731455]
  Deep learning (DL) based computer vision (CV) models are generally considered as black boxes due to poor interpretability. This study is motivated by the need to enhance the interpretability of DL model in autonomous driving. The proposed framework imitates the learning process of human drivers by jointly modeling the visual input (images) and natural language.
  arXiv  Detail & Related papers  (2021-10-11T18:50:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.