In the Driver's Seat: A Multi-Company Study on the Reality of Autonomous Driving System Testing

• URL: http://arxiv.org/abs/2607.15820v1
• Date: Fri, 17 Jul 2026 10:28:42 GMT
• Title: In the Driver's Seat: A Multi-Company Study on the Reality of Autonomous Driving System Testing
• Abstract summary: Autonomous driving systems (ADS) are rapidly advancing and increasingly deployed in real-world applications.<n>ADS testing remains complex and lacks well-established standards for scenario selection, performance evaluation, and acceptance criteria.<n>This study conducted an interview study with experts working on ADS development and testing in nine companies from six different countries.
• Score: 9.65830001261678
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous driving systems (ADS) are rapidly advancing and increasingly deployed in real-world applications. This creates growing demands for effective testing to ensure system functionality and safety. However, ADS testing remains complex and lacks well-established standards for scenario selection, performance evaluation, and acceptance criteria. To better understand current ADS testing practices and challenges, we conducted an interview study with experts working on ADS development and testing in nine companies from six different countries. Through thematic analysis, we synthesized industrial testing practices, challenges, potential solutions, future trends, and proposed an evidence-centered closed-loop testing framework for ADS testing. Our findings show that current practices primarily focus on scenario-based and X-in-the-loop testing approaches, supported by diverse tools, metrics, benchmarks, and testing strategies. The participants highlighted major challenges related to scenario realism, scenario coverage, simulation fidelity, and acceptance criteria, while also discussing potential solutions such as the use of AI, world models, and end-to-end approaches. Furthermore, participants envisioned future ADS testing to become more automated, data-driven, and transparent across the industry. Overall, this study provides a comprehensive industry-grounded overview of ADS testing, proposes an evidence-centered closed-loop testing framework to provide actionable guidance for ADS testing, and outlines important directions for future research and practice.

Related papers

• From Research to Practice: An Interactive Rapid Review of Autonomous Driving System Testing in Industry [4.424336158797069]
  Autonomous driving systems (ADS) are increasingly deployed in real traffic, yet testing remains fundamentally challenging.<n>Despite extensive research, a gap persists between academic advances and their applicability in industrial practice.<n>This study provides the first practitioner-driven review and evaluation of current ADS testing research.
  arXiv  Detail & Related papers  (2026-05-01T09:13:03Z)
• Let the Barbarians In: How AI Can Accelerate Systems Performance Research [80.43506848683633]
  We term this iterative cycle of generation, evaluation, and refinement AI-Driven Research for Systems.<n>We demonstrate that ADRS-generated solutions can match or even outperform human state-of-the-art designs.
  arXiv  Detail & Related papers  (2025-12-16T18:51:23Z)
• Advancing Autonomous Driving System Testing: Demands, Challenges, and Future Directions [16.701909787572724]
  Autonomous driving systems (ADSs) promise improved transportation efficiency and safety, yet ensuring their reliability in complex real-world environments remains a critical challenge.<n>This study investigates current ADS testing practices for both modular and end-to-end systems.<n>We review major testing techniques and consider emerging factors such as Vehicle-to-Everything (V2X) communication and foundation models.
  arXiv  Detail & Related papers  (2025-12-09T06:33:27Z)
• VP-AutoTest: A Virtual-Physical Fusion Autonomous Driving Testing Platform [47.03629760732478]
  We propose the Virtual-Physical Testing Platform for Autonomous Vehicles (VP-AutoTest) to replicate the diversity of real-world traffic participants.<n>VP-AutoTest incorporates a multidimensional evaluation framework and AI-driven expert systems to conduct comprehensive performance assessment and defect diagnosis.
  arXiv  Detail & Related papers  (2025-12-08T12:43:33Z)
• Generative AI for Testing of Autonomous Driving Systems: A Survey [13.226510198306885]
  Autonomous driving systems (ADS) have been an active area of research, with the potential to deliver significant benefits to society.<n>Different testing approaches are required, and achieving effective and efficient testing of ADS remains an open challenge.<n>generative AI has emerged as a powerful tool across many domains, and it is increasingly being applied to ADS testing due to its ability to interpret context.<n>This survey provides an overview and practical insights into the use of generative AI for testing ADS, highlights existing challenges, and outlines directions for future research in this rapidly evolving field.
  arXiv  Detail & Related papers  (2025-08-27T13:40:14Z)
• TestAgent: An Adaptive and Intelligent Expert for Human Assessment [62.060118490577366]
  We propose TestAgent, a large language model (LLM)-powered agent designed to enhance adaptive testing through interactive engagement.<n>TestAgent supports personalized question selection, captures test-takers' responses and anomalies, and provides precise outcomes through dynamic, conversational interactions.
  arXiv  Detail & Related papers  (2025-06-03T16:07:54Z)
• Fine-grained Testing for Autonomous Driving Software: a Study on Autoware with LLM-driven Unit Testing [12.067489008051208]
  We present the first study on testing, specifically unit testing, for autonomous driving systems (ADS) source code.<n>We analyze both human-written test cases and those generated by large language models (LLMs)<n>We propose AwTest-LLM, a novel approach to enhance test coverage and improve test case pass rates across Autoware packages.
  arXiv  Detail & Related papers  (2025-01-16T22:36:00Z)
• Survey of Computerized Adaptive Testing: A Machine Learning Perspective [66.26687542572974]
  Computerized Adaptive Testing (CAT) provides an efficient and tailored method for assessing the proficiency of examinees. This paper aims to provide a machine learning-focused survey on CAT, presenting a fresh perspective on this adaptive testing method.
  arXiv  Detail & Related papers  (2024-03-31T15:09:47Z)
• Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness [53.91018508439669]
  The study explores the complexities of integrating Artificial Intelligence into Autonomous Vehicles (AVs) It examines the challenges introduced by AI components and the impact on testing procedures. The paper identifies significant challenges and suggests future directions for research and development of AI in AV technology.
  arXiv  Detail & Related papers  (2024-02-21T08:29:42Z)
• A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation [26.10081199009559]
  Testing the system on the road is the closest to real-world and desirable approach, but it is incredibly costly. A popular alternative is to evaluate an ADS's performance in some well-designed challenging scenarios, a.k.a. scenario-based testing. High-fidelity simulators have been widely used in this setting to maximize flexibility and convenience in testing what-if scenarios.
  arXiv  Detail & Related papers  (2021-12-02T03:41:33Z)
• Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework [68.96770035057716]
  A/B testing is a business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries. This paper introduces a reinforcement learning framework for carrying A/B testing in online experiments.
  arXiv  Detail & Related papers  (2020-02-05T10:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.