Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses

• URL: http://arxiv.org/abs/2302.08418v1
• Date: Thu, 16 Feb 2023 16:54:10 GMT
• Title: Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses
• Authors: Minrui Xu, Dusit Niyato, Junlong Chen, Hongliang Zhang, Jiawen Kang, Zehui Xiong, Shiwen Mao, Zhu Han
• Abstract summary: In vehicular mixed reality (MR) Metaverse, distance between physical and virtual entities can be overcome. Large-scale traffic and driving simulation via realistic data collection and fusion from the physical world is difficult and costly. We propose an autonomous driving architecture, where generative AI is leveraged to synthesize unlimited conditioned traffic and driving data in simulations.
• Score: 130.15554653948897
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the vehicular mixed reality (MR) Metaverse, the distance between physical and virtual entities can be overcome by fusing the physical and virtual environments with multi-dimensional communications in autonomous driving systems. Assisted by digital twin (DT) technologies, connected autonomous vehicles (AVs), roadside units (RSU), and virtual simulators can maintain the vehicular MR Metaverse via digital simulations for sharing data and making driving decisions collaboratively. However, large-scale traffic and driving simulation via realistic data collection and fusion from the physical world for online prediction and offline training in autonomous driving systems are difficult and costly. In this paper, we propose an autonomous driving architecture, where generative AI is leveraged to synthesize unlimited conditioned traffic and driving data in simulations for improving driving safety and traffic efficiency. First, we propose a multi-task DT offloading model for the reliable execution of heterogeneous DT tasks with different requirements at RSUs. Then, based on the preferences of AV's DTs and collected realistic data, virtual simulators can synthesize unlimited conditioned driving and traffic datasets to further improve robustness. Finally, we propose a multi-task enhanced auction-based mechanism to provide fine-grained incentives for RSUs in providing resources for autonomous driving. The property analysis and experimental results demonstrate that the proposed mechanism and architecture are strategy-proof and effective, respectively.

Related papers

• A Joint Approach Towards Data-Driven Virtual Testing for Automated Driving: The AVEAS Project [2.4163276807189282]
  There is a significant shortage of real-world data to parametrize and/or validate simulations. This paper presents the results of the German AVEAS research project.
  arXiv  Detail & Related papers  (2024-05-10T07:36:03Z)
• An Approach to Systematic Data Acquisition and Data-Driven Simulation for the Safety Testing of Automated Driving Functions [32.37902846268263]
  In R&D areas related to the safety impact of the "open world", there is a significant shortage of real-world data to parameterize and/or validate simulations. We present an approach to systematically acquire data in public traffic by heterogeneous means, transform it into a unified representation, and use it to automatically parameterize traffic behavior models for use in data-driven virtual validation of automated driving functions.
  arXiv  Detail & Related papers  (2024-05-02T23:24:27Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• A Multi-Modal Simulation Framework to Enable Digital Twin-based V2X Communications in Dynamic Environments [10.652127049174883]
  Digital Twins (DTs) for physical wireless environments have been recently proposed as accurate virtual representations of the propagation environment. We propose a novel data-driven workflow for the creation of the DT of a Vehicle-to-Everything (V2X) communication scenario. We showcase the proposed framework on the DT-aided blockage handover task for V2X link restoration.
  arXiv  Detail & Related papers  (2023-03-13T09:31:20Z)
• TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction [149.5716746789134]
  We show data-driven traffic simulation can be formulated as a world model. We present TrafficBots, a multi-agent policy built upon motion prediction and end-to-end driving. Experiments on the open motion dataset show TrafficBots can simulate realistic multi-agent behaviors.
  arXiv  Detail & Related papers  (2023-03-07T18:28:41Z)
• Generative AI-empowered Effective Physical-Virtual Synchronization in the Vehicular Metaverse [129.8037449161817]
  We propose a generative AI-empowered physical-virtual synchronization framework for the vehicular Metaverse. In virtual-to-physical synchronization, MARs customize diverse and personal AR recommendations via generative AI models based on user preferences.
  arXiv  Detail & Related papers  (2023-01-18T16:25:42Z)
• Exiting the Simulation: The Road to Robust and Resilient Autonomous Vehicles at Scale [0.0]
  This paper presents the current state-of-the-art simulation frameworks and methodologies used in the development of autonomous driving systems. A synthesis of the key challenges surrounding autonomous driving simulation is presented.
  arXiv  Detail & Related papers  (2022-10-19T20:32:43Z)
• CARNet: A Dynamic Autoencoder for Learning Latent Dynamics in Autonomous Driving Tasks [11.489187712465325]
  An autonomous driving system should effectively use the information collected from the various sensors in order to form an abstract description of the world. Deep learning models, such as autoencoders, can be used for that purpose, as they can learn compact latent representations from a stream of incoming data. This work proposes CARNet, a Combined dynAmic autoencodeR NETwork architecture that utilizes an autoencoder combined with a recurrent neural network to learn the current latent representation.
  arXiv  Detail & Related papers  (2022-05-18T04:15:42Z)
• COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehicles [54.61668577827041]
  We introduce COOPERNAUT, an end-to-end learning model that uses cross-vehicle perception for vision-based cooperative driving. Our experiments on AutoCastSim suggest that our cooperative perception driving models lead to a 40% improvement in average success rate.
  arXiv  Detail & Related papers  (2022-05-04T17:55:12Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.