DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models

• URL: http://arxiv.org/abs/2606.17362v1
• Date: Mon, 15 Jun 2026 23:39:36 GMT
• Title: DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models
• Abstract summary: DriveJudge is a driving evaluation agent that combines rule-grounded evaluation with Vision-Language Model (VLM) reasoning.<n>We introduce two benchmark tasks: Driving Quality Classification and Trajectory Preference Selection.<n>DriveJudge outperformsS for driving quality classification by 21.23 AUC, and the recent VLM-based DriveCritic for trajectory preference selection by 6.5%.
• Score: 53.73898294218455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous driving has shifted towards end-to-end policy learning, where reliable, interpretable policy evaluation is a fundamental challenge as driving quality is highly context-dependent. Commonly used rule-based driving metrics like EPDMS are interpretable but lack context-awareness, while recent VLMbased evaluations are context-aware but limited by ambiguous VLM outputs and weak physical grounding. To evaluate driving in a manner that is both interpretable and context-aware, we introduce DriveJudge. DriveJudge is a driving evaluation agent that combines rule-grounded evaluation with Vision-Language Model (VLM) reasoning and selectively invokes physically-grounded deterministic rule functions after interpreting the environmental context. To train and evaluate DriveJudge, we curate a large-scale dataset of 33,577 challenging driving samples with human annotations on whether the driving behavior is reasonable in the given scenario. With this dataset, we address the underexplored problem of driving metric evaluation, and introduce two human-aligned benchmark tasks: Driving Quality Classification and Trajectory Preference Selection. DriveJudge outperforms EPDMS for driving quality classification by 21.23 AUC, and the recent VLM-based DriveCritic for trajectory preference selection by 6.5%, setting a new standard for interpretable and precise driving evaluation.

Related papers

• Beyond Fixed Thresholds and Domain-Specific Benchmarks for Explainable Multi-Task Classification in Autonomous Vehicles [0.0]
  Scene understanding is a vital part of autonomous driving systems, which requires the use of deep learning models.<n>We introduce a confidence threshold sensitivity analysis that evaluates various threshold values to identify optimal decision boundaries for different tasks.<n>We introduce IUST-XAI-AD, a novel dataset consisting of 958 images with human annotations for driving decisions and corresponding reasoning.
  arXiv  Detail & Related papers  (2026-05-05T21:07:11Z)
• Can Users Specify Driving Speed? Bench2Drive-Speed: Benchmark and Baselines for Desired-Speed Conditioned Autonomous Driving [50.09883181221131]
  We present Bench2Drive-Speed, a benchmark with metrics, dataset, and baselines for desired-speed conditioned autonomous driving.<n>We introduce explicit inputs of users' desired target-speed and overtake/follow instructions to driving policy models.<n>Our experiments show that, under proper re-annotation, models trained on regular driving data perform comparably to on expert demonstrations.
  arXiv  Detail & Related papers  (2026-03-26T17:26:41Z)
• SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios [104.10555123175055]
  A fundamental challenge in autonomous driving is the integration of high-level, semantic reasoning for long-tail events with low-level, reactive control for robust driving.<n>We propose SteerVLA, which produces fine-grained language instructions that steer a vision-language-action driving policy.<n>We evaluate SteerVLA on a challenging closed-loop benchmark, where it outperforms state-of-the-art methods by 4.77 points in overall driving score and by 8.04 points on a long-tail subset.
  arXiv  Detail & Related papers  (2026-02-09T09:54:02Z)
• Measuring What Matters: Scenario-Driven Evaluation for Trajectory Predictors in Autonomous Driving [8.115144579497644]
  We propose a comprehensive pipeline that adaptively evaluates the predictor's performance by two dimensions: accuracy and diversity.<n>Experiments on a closed-loop benchmark using real-world datasets show that our pipeline yields a more reasonable evaluation than traditional metrics.
  arXiv  Detail & Related papers  (2025-12-13T06:48:32Z)
• DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models [24.168614747778538]
  We introduce DriveCritic, a novel framework featuring two key contributions.<n>The dataset is a curated collection of challenging scenarios where context is critical for correct judgment.<n>The DriveCritic model learns to adjudicate between trajectory pairs by integrating visual and symbolic context.
  arXiv  Detail & Related papers  (2025-10-15T03:00:38Z)
• DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models [5.1501039725906885]
  Vision-Language-Action (VLA) models have advanced autonomous driving, but existing benchmarks still lack scenario diversity, reliable action-level annotation, and evaluation protocols aligned with human preferences.<n>We introduce DriveAction, the first action-driven benchmark specifically designed for VLA models, comprising 16,185 QA pairs generated from 2,610 driving scenarios.<n>Our experiments demonstrate that state-of-the-art vision-language models (VLMs) require both vision and language guidance for accurate action prediction.
  arXiv  Detail & Related papers  (2025-06-06T01:30:52Z)
• PADriver: Towards Personalized Autonomous Driving [27.96579880234604]
  We propose PADriver, a novel closed-loop framework for personalized autonomous driving (PAD)<n>Built upon Multi-modal Large Language Model (MLLM), PADriver takes streaming frames and personalized textual prompts as inputs.<n>We construct a benchmark named PAD-Highway based on Highway-Env simulator to comprehensively evaluate the decision performance under traffic rules.
  arXiv  Detail & Related papers  (2025-05-08T13:36:07Z)
• Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives [56.528835143531694]
  We introduce DriveBench, a benchmark dataset designed to evaluate Vision-Language Models (VLMs)<n>Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding.<n>We propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding.
  arXiv  Detail & Related papers  (2025-01-07T18:59:55Z)
• Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases [102.05741859030951]
  We propose CODA-LM, the first benchmark for the automatic evaluation of LVLMs for self-driving corner cases.<n>We show that using the text-only large language models as judges reveals even better alignment with human preferences than the LVLM judges.<n>Our CODA-VLM performs comparably with GPT-4V, even surpassing GPT-4V by +21.42% on the regional perception task.
  arXiv  Detail & Related papers  (2024-04-16T14:20:55Z)
• DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving [81.04174379726251]
  This paper collects a comprehensive end-to-end driving dataset named DriveCoT. It contains sensor data, control decisions, and chain-of-thought labels to indicate the reasoning process. We propose a baseline model called DriveCoT-Agent, trained on our dataset, to generate chain-of-thought predictions and final decisions.
  arXiv  Detail & Related papers  (2024-03-25T17:59:01Z)
• Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts? [104.04999499189402]
  Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment. We propose an uncertainty-aware planning method, called emphrobust imitative planning (RIP) Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. We introduce an autonomous car novel-scene benchmark, textttCARNOVEL, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.
  arXiv  Detail & Related papers  (2020-06-26T11:07:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.