AutoDrive-QA- Automated Generation of Multiple-Choice Questions for Autonomous Driving Datasets Using Large Vision-Language Models

• URL: http://arxiv.org/abs/2503.15778v1
• Date: Thu, 20 Mar 2025 01:32:00 GMT
• Title: AutoDrive-QA- Automated Generation of Multiple-Choice Questions for Autonomous Driving Datasets Using Large Vision-Language Models
• Authors: Boshra Khalili, Andrew W. Smyth,
• Abstract summary: We introduce AutoDrive-QA, an automatic pipeline that converts existing driving QA datasets into a structured multiple-choice question (MCQ) format.<n>This benchmark systematically assesses perception, prediction, and planning tasks, providing a standardized and objective evaluation framework.<n>We test the benchmark on three public datasets and conduct zero-shot experiments on an unseen dataset.
• Score: 1.3812010983144802
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In autonomous driving, open-ended question answering often suffers from unreliable evaluations because freeform responses require either complex metrics or subjective human judgment. To address this challenge, we introduce AutoDrive-QA, an automatic pipeline that converts existing driving QA datasets (including DriveLM, NuScenes-QA, and LingoQA) into a structured multiple-choice question (MCQ) format. This benchmark systematically assesses perception, prediction, and planning tasks, providing a standardized and objective evaluation framework. AutoDrive-QA employs an automated pipeline that leverages large language models (LLMs) to generate high-quality, contextually relevant distractors based on domain-specific error patterns commonly found in autonomous driving scenarios. To evaluate both general capabilities and generalization performance, we test the benchmark on three public datasets and conduct zero-shot experiments on an unseen dataset. The zero-shot evaluations reveal that GPT-4V leads with 69.57% accuracy -- achieving 74.94% in Perception, 65.33% in Prediction, and 68.45% in Planning -- demonstrating that while all models excel in Perception, they struggle in Prediction. Consequently, AutoDrive-QA establishes a rigorous, unbiased standard for integrating and evaluating different vision-language models across various autonomous driving datasets, thereby improving generalization in this field. We release all the codes in the AutoDrive-QA GitHub Repository.

Related papers

• DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding [76.3876070043663]
  We propose DriveLMM-o1, a dataset and benchmark designed to advance step-wise visual reasoning for autonomous driving.<n>Our benchmark features over 18k VQA examples in the training set and more than 4k in the test set, covering diverse questions on perception, prediction, and planning.<n>Our model achieves a +7.49% gain in final answer accuracy, along with a 3.62% improvement in reasoning score over the previous best open-source model.
  arXiv  Detail & Related papers  (2025-03-13T17:59:01Z)
• Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation [69.81654421834989]
  We introduce Auto, an agentic framework that automatically converts open-ended questions into multiple-choice format.<n>Using Auto, we construct VMCBench, a benchmark created by transforming 20 existing VQA datasets into a unified multiple-choice format.<n>We evaluate 33 state-of-the-art vision language models on VMCBench, setting a new standard for scalable, consistent, and reproducible VLM evaluation.
  arXiv  Detail & Related papers  (2025-01-06T18:57:31Z)
• DriveMM: All-in-One Large Multimodal Model for Autonomous Driving [63.882827922267666]
  DriveMM is a large multimodal model designed to process diverse data inputs, such as images and multi-view videos, while performing a broad spectrum of autonomous driving tasks. We conduct evaluations on six public benchmarks and undertake zero-shot transfer on an unseen dataset, where DriveMM achieves state-of-the-art performance across all tasks.
  arXiv  Detail & Related papers  (2024-12-10T17:27:32Z)
• AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving [68.73885845181242]
  We propose an Automatic Data Engine (AIDE) that automatically identifies issues, efficiently curates data, improves the model through auto-labeling, and verifies the model through generation of diverse scenarios. We further establish a benchmark for open-world detection on AV datasets to comprehensively evaluate various learning paradigms, demonstrating our method's superior performance at a reduced cost.
  arXiv  Detail & Related papers  (2024-03-26T04:27:56Z)
• AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning [54.47116888545878]
  AutoAct is an automatic agent learning framework for QA. It does not rely on large-scale annotated data and synthetic planning trajectories from closed-source models.
  arXiv  Detail & Related papers  (2024-01-10T16:57:24Z)
• DriveLM: Driving with Graph Visual Question Answering [57.51930417790141]
  We study how vision-language models (VLMs) trained on web-scale data can be integrated into end-to-end driving systems.<n>We propose a VLM-based baseline approach (DriveLM-Agent) for jointly performing Graph VQA and end-to-end driving.
  arXiv  Detail & Related papers  (2023-12-21T18:59:12Z)
• NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations [0.6827423171182154]
  Visual Question Answering (VQA) is one of the most important tasks in autonomous driving. We introduce a novel dataset annotation technique in which QAs are enclosed within markups. This dataset empowers the development of vision language models, especially for autonomous driving tasks.
  arXiv  Detail & Related papers  (2023-12-11T12:58:54Z)
• Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving [38.28159034562901]
  Reason2Drive is a benchmark dataset with over 600K video-text pairs. We characterize the autonomous driving process as a sequential combination of perception, prediction, and reasoning steps. We introduce a novel aggregated evaluation metric to assess chain-based reasoning performance in autonomous systems.
  arXiv  Detail & Related papers  (2023-12-06T18:32:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.