A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems

• URL: http://arxiv.org/abs/2509.22379v2
• Date: Mon, 29 Sep 2025 07:56:47 GMT
• Title: A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems
• Authors: Stefano Carlo Lambertenghi, Mirena Flores Valdez, Andrea Stocco,
• Abstract summary: We compare four representative testing modalities: Software-in-the-Loop (SiL), Vehicle-in-the-Loop (ViL), Mixed-Reality (MR) and full real-world testing.<n>Our results show that while SiL and ViL setups simplify critical aspects of real-world dynamics and sensing, MR testing improves perceptual realism without compromising safety or control.
• Score: 0.9956658791307307
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Simulation-based testing is a cornerstone of Autonomous Driving System (ADS) development, offering safe and scalable evaluation across diverse driving scenarios. However, discrepancies between simulated and real-world behavior, known as the reality gap, challenge the transferability of test results to deployed systems. In this paper, we present a comprehensive empirical study comparing four representative testing modalities: Software-in-the-Loop (SiL), Vehicle-in-the-Loop (ViL), Mixed-Reality (MR), and full real-world testing. Using a small-scale physical vehicle equipped with real sensors (camera and LiDAR) and its digital twin, we implement each setup and evaluate two ADS architectures (modular and end-to-end) across diverse indoor driving scenarios involving real obstacles, road topologies, and indoor environments. We systematically assess the impact of each testing modality along three dimensions of the reality gap: actuation, perception, and behavioral fidelity. Our results show that while SiL and ViL setups simplify critical aspects of real-world dynamics and sensing, MR testing improves perceptual realism without compromising safety or control. Importantly, we identify the conditions under which failures do not transfer across testing modalities and isolate the underlying dimensions of the gap responsible for these discrepancies. Our findings offer actionable insights into the respective strengths and limitations of each modality and outline a path toward more robust and transferable validation of autonomous driving systems.

Related papers

• Deep Learning-Based Object Detection for Autonomous Vehicles: A Comparative Study of One-Stage and Two-Stage Detectors on Basic Traffic Objects [0.0]
  This study compares two object detection models: YOLOv5 and Faster R-CNN.<n>YOLOv5 shows superior performance in terms of mAP, recall, and training efficiency.<n>However, Faster R-CNN shows advantages in detecting small, distant objects.
  arXiv  Detail & Related papers  (2026-01-30T23:05:13Z)
• Assessing behaviour coverage in a multi-agent system simulation for autonomous vehicle testing [0.0]
  This study focuses on the behaviour coverage analysis of a multi-agent system simulation designed for autonomous vehicle testing.<n>By defining a set of driving scenarios, and agent interactions, we evaluate the extent to which the simulation encompasses a broad range of behaviours relevant to autonomous driving.
  arXiv  Detail & Related papers  (2025-12-29T13:02:32Z)
• LLM-based Human-like Traffic Simulation for Self-driving Tests [16.752944112972667]
  We introduce HDSim, an HD traffic generation framework that combines cognitive theory with large language model (LLM) assistance to produce realistic traffic scenarios.<n>Experiments reveal that embedding HDSim into simulation improves detection of safety-critical failures in self-driving systems by up to 68% and yields realism-consistent accident interpretability.
  arXiv  Detail & Related papers  (2025-08-23T09:30:55Z)
• MetAdv: A Unified and Interactive Adversarial Testing Platform for Autonomous Driving [63.875372281596576]
  MetAdv is a novel adversarial testing platform that enables realistic, dynamic, and interactive evaluation.<n>It supports flexible 3D vehicle modeling and seamless transitions between simulated and physical environments.<n>It enables real-time capture of physiological signals and behavioral feedback from drivers.
  arXiv  Detail & Related papers  (2025-08-04T03:07:54Z)
• RealEngine: Simulating Autonomous Driving in Realistic Context [60.55873455475112]
  RealEngine is a novel driving simulation framework that holistically integrates 3D scene reconstruction and novel view synthesis techniques.<n>By leveraging real-world multi-modal sensor data, RealEngine reconstructs background scenes and foreground traffic participants separately, allowing for highly diverse and realistic traffic scenarios.<n>RealEngine supports three essential driving simulation categories: non-reactive simulation, safety testing, and multi-agent interaction.
  arXiv  Detail & Related papers  (2025-05-22T17:01:00Z)
• TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learning [61.33599727106222]
  TeLL-Drive is a hybrid framework that integrates a Teacher LLM to guide an attention-based Student DRL policy.<n>A self-attention mechanism then fuses these strategies with the DRL agent's exploration, accelerating policy convergence and boosting robustness.
  arXiv  Detail & Related papers  (2025-02-03T14:22:03Z)
• Behavioural gap assessment of human-vehicle interaction in real and virtual reality-based scenarios in autonomous driving [7.588679613436823]
  We present a first and innovative approach to evaluating what we term the behavioural gap, a concept that captures the disparity in a participant's conduct when engaging in a VR experiment compared to an equivalent real-world situation. In the experiment, the pedestrian attempts to cross the road in the presence of different driving styles and an external Human-Machine Interface (eHMI) Results show that participants are more cautious and curious in VR, affecting their speed and decisions, and that VR interfaces significantly influence their actions.
  arXiv  Detail & Related papers  (2024-07-04T17:20:17Z)
• Probing Multimodal LLMs as World Models for Driving [72.18727651074563]
  We look at the application of Multimodal Large Language Models (MLLMs) in autonomous driving. Despite advances in models like GPT-4o, their performance in complex driving environments remains largely unexplored.
  arXiv  Detail & Related papers  (2024-05-09T17:52:42Z)
• Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing [2.194575078433007]
  Simulation-based testing of automated driving systems (ADS) is the industry standard, being a controlled, safe, and cost-effective alternative to real-world testing. Despite these advantages, virtual simulations often fail to accurately replicate real-world conditions like image fidelity, texture representation, and environmental accuracy. This can lead to significant differences in ADS behavior between simulated and real-world domains, a phenomenon known as the sim2real gap. Researchers have used Image-to-Image (I2I) neural translation to mitigate the sim2real gap, enhancing the realism of simulated environments by transforming synthetic data into more authentic
  arXiv  Detail & Related papers  (2024-04-29T10:37:38Z)
• First-principles Based 3D Virtual Simulation Testing for Discovering SOTIF Corner Cases of Autonomous Driving [5.582213904792781]
  This paper proposes a first-principles based sensor modeling and environment interaction scheme, and integrates it into CARLA simulator. A meta-heuristic algorithm is designed based on several empirical insights, which guide both seed scenarios and mutations. Under identical simulation setups, our algorithm discovers about four times as many corner cases as compared to state-of-the-art work.
  arXiv  Detail & Related papers  (2024-01-22T12:02:32Z)
• Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses [130.15554653948897]
  In vehicular mixed reality (MR) Metaverse, distance between physical and virtual entities can be overcome. Large-scale traffic and driving simulation via realistic data collection and fusion from the physical world is difficult and costly. We propose an autonomous driving architecture, where generative AI is leveraged to synthesize unlimited conditioned traffic and driving data in simulations.
  arXiv  Detail & Related papers  (2023-02-16T16:54:10Z)
• NeurIPS 2022 Competition: Driving SMARTS [60.948652154552136]
  Driving SMARTS is a regular competition designed to tackle problems caused by the distribution shift in dynamic interaction contexts. The proposed competition supports methodologically diverse solutions, such as reinforcement learning (RL) and offline learning methods.
  arXiv  Detail & Related papers  (2022-11-14T17:10:53Z)
• Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems [6.649715954440713]
  We leverage the Donkey Car open-source framework to empirically compare testing of SDCs when deployed on a physical small-scale vehicle vs its virtual simulated counterpart. While a large number of testing results do transfer between virtual and physical environments, we also identified critical shortcomings that contribute to the reality gap between the virtual and physical world.
  arXiv  Detail & Related papers  (2021-12-21T14:28:35Z)
• From Simulation to Real World Maneuver Execution using Deep Reinforcement Learning [69.23334811890919]
  Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios. This is mainly due to the lack of domain adaptation between simulated and real-world data together with the absence of distinction between train and test datasets. We present a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios.
  arXiv  Detail & Related papers  (2020-05-13T14:22:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.