A Mixed-Reality Testbed for Autonomous Vehicles

• URL: http://arxiv.org/abs/2606.19267v1
• Date: Wed, 17 Jun 2026 16:43:33 GMT
• Title: A Mixed-Reality Testbed for Autonomous Vehicles
• Abstract summary: We propose a mixed-reality, hardware-in-the-loop testbed for autonomous vehicles.<n>The testbed seamlessly integrates a physical testbed of mobile robots with a high-fidelity simulation environment.<n>We present a safety-guaranteed framework combining perception, planning and a novel online learning-based controller.
• Score: 5.831829846656395
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a mixed-reality, hardware-in-the-loop (HIL) testbed for autonomous vehicles that seamlessly integrates a physical testbed of mobile robots with a high-fidelity simulation environment. The virtual simulation enables the creation of diverse, safety-critical driving scenarios to validate state-of-the-art perception, planning, and control algorithms, while augmenting simulations with physical robots equipped with multimodal sensors in photorealistic virtual environments further facilitating rigorous validation. Our testbed also features vehicular connectivity using wireless communication and can accommodate a large number of agents through the combination of physical robots and virtual simulated agents, supporting research on multi-agent systems including Connected and Autonomous Vehicles (CAVs). Finally, we present a safety-guaranteed framework combining perception, planning and a novel online learning-based controller using Control Barrier Functions (CBFs) for CAVs. Experiments using the proposed framework are used to validate and demonstrate the key functionalities and the overall utility of the testbed to bridge the gap between simulation and real-world hardware deployment.

Related papers

• IR-SIM: A Lightweight Skill-Native Simulator for Navigation, Learning, and Benchmarking [59.6657632820611]
  IR-SIM is a skill-native navigation simulator designed for rapid scenario construction, benchmarking, and robot learning.<n> scenarios are entirely defined by YAML configuration files that specify mobile robot kinematics, geometric collision checking, LiDAR sensing, visualization, and behavior modules.<n>The experiments showcase the convenience and versatility of IR-SIM in multiple tasks.
  arXiv  Detail & Related papers  (2026-06-07T16:55:37Z)
• Closed-Loop Hybrid Digital Twin Platform for Connected and Automated Vehicle Validation [8.373850248021927]
  This paper proposes a novel real-time hybrid digital twin platform for connected and automated vehicles (CAVs) validation.<n>Its core innovation lies in the tight coupling of a high-fidelity CARLA-SUMO co-simulation with a physical test site and vehicle via a low-latency Vehicle-to-Everything (V2X) communication link.<n> Detailed implementation includes using photogrammetry for full-scale asset reconstruction and a cloud-edge collaborative architecture for scalable, multi-user operation.
  arXiv  Detail & Related papers  (2026-05-19T07:44:59Z)
• From Code to Road: A Vehicle-in-the-Loop and Digital Twin-Based Framework for Central Car Server Testing in Autonomous Driving [12.14436913195805]
  This work presents a test framework for automotive software on the centralized E/E architecture.<n>The framework couples a physical test vehicle on a dynamometer test bench with its synchronized virtual counterpart in a simulation environment.
  arXiv  Detail & Related papers  (2026-03-05T15:23:28Z)
• MetAdv: A Unified and Interactive Adversarial Testing Platform for Autonomous Driving [85.04826012938642]
  MetAdv is a novel adversarial testing platform that enables realistic, dynamic, and interactive evaluation.<n>It supports flexible 3D vehicle modeling and seamless transitions between simulated and physical environments.<n>It enables real-time capture of physiological signals and behavioral feedback from drivers.
  arXiv  Detail & Related papers  (2025-08-04T03:07:54Z)
• RealEngine: Simulating Autonomous Driving in Realistic Context [60.55873455475112]
  RealEngine is a novel driving simulation framework that holistically integrates 3D scene reconstruction and novel view synthesis techniques.<n>By leveraging real-world multi-modal sensor data, RealEngine reconstructs background scenes and foreground traffic participants separately, allowing for highly diverse and realistic traffic scenarios.<n>RealEngine supports three essential driving simulation categories: non-reactive simulation, safety testing, and multi-agent interaction.
  arXiv  Detail & Related papers  (2025-05-22T17:01:00Z)
• A Joint Approach Towards Data-Driven Virtual Testing for Automated Driving: The AVEAS Project [2.4163276807189282]
  There is a significant shortage of real-world data to parametrize and/or validate simulations. This paper presents the results of the German AVEAS research project.
  arXiv  Detail & Related papers  (2024-05-10T07:36:03Z)
• Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses [130.15554653948897]
  In vehicular mixed reality (MR) Metaverse, distance between physical and virtual entities can be overcome. Large-scale traffic and driving simulation via realistic data collection and fusion from the physical world is difficult and costly. We propose an autonomous driving architecture, where generative AI is leveraged to synthesize unlimited conditioned traffic and driving data in simulations.
  arXiv  Detail & Related papers  (2023-02-16T16:54:10Z)
• Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems [6.649715954440713]
  We leverage the Donkey Car open-source framework to empirically compare testing of SDCs when deployed on a physical small-scale vehicle vs its virtual simulated counterpart. While a large number of testing results do transfer between virtual and physical environments, we also identified critical shortcomings that contribute to the reality gap between the virtual and physical world.
  arXiv  Detail & Related papers  (2021-12-21T14:28:35Z)
• VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles [131.2240621036954]
  We present VISTA, an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras. We demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle.
  arXiv  Detail & Related papers  (2021-11-23T18:58:10Z)
• Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction [88.0416857308144]
  We propose an alternative to sensor simulation, as sensor simulation is expensive and has large domain gaps. We directly simulate the outputs of the self-driving vehicle's perception and prediction system, enabling realistic motion planning testing.
  arXiv  Detail & Related papers  (2020-08-13T17:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.