GAIA-1: A Generative World Model for Autonomous Driving
- URL: http://arxiv.org/abs/2309.17080v1
- Date: Fri, 29 Sep 2023 09:20:37 GMT
- Title: GAIA-1: A Generative World Model for Autonomous Driving
- Authors: Anthony Hu and Lloyd Russell and Hudson Yeo and Zak Murez and George
Fedoseev and Alex Kendall and Jamie Shotton and Gianluca Corrado
- Abstract summary: We introduce GAIA-1 ('Generative AI for Autonomy'), a generative world model that generates realistic driving scenarios.
Emerging properties from our model include learning high-level structures and scene dynamics, contextual awareness, generalization, and understanding of geometry.
- Score: 9.578453700755318
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Autonomous driving promises transformative improvements to transportation,
but building systems capable of safely navigating the unstructured complexity
of real-world scenarios remains challenging. A critical problem lies in
effectively predicting the various potential outcomes that may emerge in
response to the vehicle's actions as the world evolves.
To address this challenge, we introduce GAIA-1 ('Generative AI for
Autonomy'), a generative world model that leverages video, text, and action
inputs to generate realistic driving scenarios while offering fine-grained
control over ego-vehicle behavior and scene features. Our approach casts world
modeling as an unsupervised sequence modeling problem by mapping the inputs to
discrete tokens, and predicting the next token in the sequence. Emerging
properties from our model include learning high-level structures and scene
dynamics, contextual awareness, generalization, and understanding of geometry.
The power of GAIA-1's learned representation that captures expectations of
future events, combined with its ability to generate realistic samples,
provides new possibilities for innovation in the field of autonomy, enabling
enhanced and accelerated training of autonomous driving technology.
Related papers
- Generalized Predictive Model for Autonomous Driving [75.39517472462089]
We introduce the first large-scale video prediction model in the autonomous driving discipline.
Our model, dubbed GenAD, handles the challenging dynamics in driving scenes with novel temporal reasoning blocks.
It can be adapted into an action-conditioned prediction model or a motion planner, holding great potential for real-world driving applications.
arXiv Detail & Related papers (2024-03-14T17:58:33Z) - GenAD: Generative End-to-End Autonomous Driving [13.332272121018285]
GenAD is a generative framework that casts autonomous driving into a generative modeling problem.
We propose an instance-centric scene tokenizer that first transforms the surrounding scenes into map-aware instance tokens.
We then employ a variational autoencoder to learn the future trajectory distribution in a structural latent space for trajectory prior modeling.
arXiv Detail & Related papers (2024-02-18T08:21:05Z) - Driving into the Future: Multiview Visual Forecasting and Planning with
World Model for Autonomous Driving [56.381918362410175]
Drive-WM is the first driving world model compatible with existing end-to-end planning models.
Our model generates high-fidelity multiview videos in driving scenes.
arXiv Detail & Related papers (2023-11-29T18:59:47Z) - Exploring the Potential of World Models for Anomaly Detection in
Autonomous Driving [11.091582432763738]
We show how world models can be leveraged to perform anomaly detection in the domain of autonomous driving.
We provide a characterization of world models and relate individual components to previous works in anomaly detection.
arXiv Detail & Related papers (2023-08-10T17:04:51Z) - Model-Based Reinforcement Learning with Isolated Imaginations [61.67183143982074]
We propose Iso-Dream++, a model-based reinforcement learning approach.
We perform policy optimization based on the decoupled latent imaginations.
This enables long-horizon visuomotor control tasks to benefit from isolating mixed dynamics sources in the wild.
arXiv Detail & Related papers (2023-03-27T02:55:56Z) - TrafficBots: Towards World Models for Autonomous Driving Simulation and
Motion Prediction [149.5716746789134]
We show data-driven traffic simulation can be formulated as a world model.
We present TrafficBots, a multi-agent policy built upon motion prediction and end-to-end driving.
Experiments on the open motion dataset show TrafficBots can simulate realistic multi-agent behaviors.
arXiv Detail & Related papers (2023-03-07T18:28:41Z) - Exiting the Simulation: The Road to Robust and Resilient Autonomous
Vehicles at Scale [0.0]
This paper presents the current state-of-the-art simulation frameworks and methodologies used in the development of autonomous driving systems.
A synthesis of the key challenges surrounding autonomous driving simulation is presented.
arXiv Detail & Related papers (2022-10-19T20:32:43Z) - Tackling Real-World Autonomous Driving using Deep Reinforcement Learning [63.3756530844707]
In this work, we propose a model-free Deep Reinforcement Learning Planner training a neural network that predicts acceleration and steering angle.
In order to deploy the system on board the real self-driving car, we also develop a module represented by a tiny neural network.
arXiv Detail & Related papers (2022-07-05T16:33:20Z) - Isolating and Leveraging Controllable and Noncontrollable Visual
Dynamics in World Models [65.97707691164558]
We present Iso-Dream, which improves the Dream-to-Control framework in two aspects.
First, by optimizing inverse dynamics, we encourage world model to learn controllable and noncontrollable sources.
Second, we optimize the behavior of the agent on the decoupled latent imaginations of the world model.
arXiv Detail & Related papers (2022-05-27T08:07:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.