Search-based Testing of Vision Language Models for In-Car Scene Understanding

• URL: http://arxiv.org/abs/2607.02300v2
• Date: Thu, 09 Jul 2026 13:51:10 GMT
• Title: Search-based Testing of Vision Language Models for In-Car Scene Understanding
• Abstract summary: In the automotive domain, in-car scene understanding (ISU) enables the detection of safety-critical events.<n>We present ISU-Test, an automated testing approach that combines rendering-based scene generation with search-based testing to evaluate ISU systems.
• Score: 2.9682153664947317
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In the automotive domain, in-car scene understanding (ISU) enables the detection of safety-critical events, such as driver distraction, and supports drivers or passengers by analyzing the in-car scene and adapting the environment (e.g., ambient lighting). The industry is increasingly exploring vision-language models (VLMs) to interpret camera-recorded in-car scenes and extract information for downstream reasoning tasks. However, VLMs may generate incomplete, erroneous, or misleading scene descriptions, highlighting the need for systematic testing. Collecting real in-vehicle data is costly, difficult to scale, and often infeasible, particularly in early design stages. In this paper, we present ISU-Test, an automated testing approach that combines rendering-based scene generation with search-based testing to evaluate ISU systems. By framing testing as an optimization problem and systematically modifying scene parameters, our method generates diverse in-car scenarios and explores a wide range of configurations. We evaluate ISU-Test on both an industrial prototype and open-source VLMs across two case studies: question answering and captioning, comparing against randomized scenario generation. Results show that ISU-Test significantly outperforms the baseline, achieving up to 10 times higher failure rates and up to 3.6 times higher failure coverage.

Related papers

• AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding [12.229867730747996]
  We present AUTOPILOT-VQA, an incident-centric visual question answering benchmark for dashcam video understanding.<n>The dataset evaluates different systems through structured questions designed around real-world driving incidents and near-incidents.<n>Our benchmark support developments for more interpretable, robust, and safety-conscious vision-language systems for real-world autonomous driving.
  arXiv  Detail & Related papers  (2026-07-09T17:46:24Z)
• CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning [27.23760411917563]
  We present textbfCrashSight, a vision-language benchmark for roadway crash understanding using real-world roadside camera data.<n>The dataset comprises 250 crash videos, annotated with 13K multiple-choice question-answer pairs organized under a two-tier taxonomy.<n>We benchmark 8 state-of-the-art VLMs and show that, despite strong scene description capabilities, current models struggle with temporal and causal reasoning in safety-critical scenarios.
  arXiv  Detail & Related papers  (2026-04-09T16:52:04Z)
• How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study [2.6954666679827137]
  VENUSS is a framework for systematic sensitivity analysis of VLM performance on sequential driving scenes.<n>It extracts temporal sequences from driving videos, and generates structured evaluations across custom categories.<n>We show how even top models achieve only 57% accuracy, not matching human performance in similar constraints.
  arXiv  Detail & Related papers  (2026-04-08T07:14:55Z)
• Overtake Detection in Trucks Using CAN Bus Signals: A Comparative Study of Machine Learning Methods [51.28632782308621]
  We focus on overtake detection using Controller Area Network (CAN) bus data collected from five in-service trucks provided by the Volvo Group.<n>We evaluate three common classifiers for vehicle manoeuvre detection, Artificial Neural Networks (ANN), Random Forest (RF), and Support Vector Machines (SVM)<n>Our pertruck analysis also reveals that classification accuracy, especially for overtakes, depends on the amount of training data per vehicle.
  arXiv  Detail & Related papers  (2025-07-01T09:20:41Z)
• Hard Cases Detection in Motion Prediction by Vision-Language Foundation Models [16.452638202694246]
  This work explores the potential of Vision-Language Foundation Models (VLMs) in detecting hard cases in autonomous driving. We introduce a feasible pipeline where VLMs, fed with sequential image frames with designed prompts, effectively identify challenging agents or scenarios. We show the effectiveness and feasibility of incorporating our pipeline with state-of-the-art methods on NuScenes datasets.
  arXiv  Detail & Related papers  (2024-05-31T16:35:41Z)
• AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving [68.73885845181242]
  We propose an Automatic Data Engine (AIDE) that automatically identifies issues, efficiently curates data, improves the model through auto-labeling, and verifies the model through generation of diverse scenarios. We further establish a benchmark for open-world detection on AV datasets to comprehensively evaluate various learning paradigms, demonstrating our method's superior performance at a reduced cost.
  arXiv  Detail & Related papers  (2024-03-26T04:27:56Z)
• DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving [76.29141888408265]
  We propose a large-scale dataset containing diverse accident scenarios that frequently occur in real-world driving. The proposed DeepAccident dataset includes 57K annotated frames and 285K annotated samples, approximately 7 times more than the large-scale nuScenes dataset.
  arXiv  Detail & Related papers  (2023-04-03T17:37:00Z)
• An Application of Scenario Exploration to Find New Scenarios for the Development and Testing of Automated Driving Systems in Urban Scenarios [2.480533141352916]
  This work aims to find relevant, interesting, or critical parameter sets within logical scenarios by utilizing Bayes optimization and Gaussian processes. A list of ideas this work leads to and should be investigated further is presented.
  arXiv  Detail & Related papers  (2022-05-17T09:47:32Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation [77.60050239225086]
  We propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images. Our approach is fully automatic without any human interaction. We present a multi-task network for VUS parsing and a multi-stream network for VHI parsing.
  arXiv  Detail & Related papers  (2020-12-15T03:03:38Z)
• VehicleNet: Learning Robust Visual Representation for Vehicle Re-identification [116.1587709521173]
  We propose to build a large-scale vehicle dataset (called VehicleNet) by harnessing four public vehicle datasets. We design a simple yet effective two-stage progressive approach to learning more robust visual representation from VehicleNet. We achieve the state-of-art accuracy of 86.07% mAP on the private test set of AICity Challenge.
  arXiv  Detail & Related papers  (2020-04-14T05:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.