A Language Agent for Autonomous Driving
- URL: http://arxiv.org/abs/2311.10813v3
- Date: Mon, 27 Nov 2023 20:53:35 GMT
- Title: A Language Agent for Autonomous Driving
- Authors: Jiageng Mao and Junjie Ye and Yuxi Qian and Marco Pavone and Yue Wang
- Abstract summary: We propose a paradigm shift to integrate human-like intelligence into autonomous driving systems.
Our approach, termed Agent-Driver, transforms the traditional autonomous driving pipeline.
Our approach significantly outperforms the state-of-the-art driving methods by a large margin.
- Score: 33.64382018350317
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Human-level driving is an ultimate goal of autonomous driving. Conventional
approaches formulate autonomous driving as a perception-prediction-planning
framework, yet their systems do not capitalize on the inherent reasoning
ability and experiential knowledge of humans. In this paper, we propose a
fundamental paradigm shift from current pipelines, exploiting Large Language
Models (LLMs) as a cognitive agent to integrate human-like intelligence into
autonomous driving systems. Our approach, termed Agent-Driver, transforms the
traditional autonomous driving pipeline by introducing a versatile tool library
accessible via function calls, a cognitive memory of common sense and
experiential knowledge for decision-making, and a reasoning engine capable of
chain-of-thought reasoning, task planning, motion planning, and
self-reflection. Powered by LLMs, our Agent-Driver is endowed with intuitive
common sense and robust reasoning capabilities, thus enabling a more nuanced,
human-like approach to autonomous driving. We evaluate our approach on the
large-scale nuScenes benchmark, and extensive experiments substantiate that our
Agent-Driver significantly outperforms the state-of-the-art driving methods by
a large margin. Our approach also demonstrates superior interpretability and
few-shot learning ability to these methods. Code will be released.
Related papers
- Exploring the Causality of End-to-End Autonomous Driving [57.631400236930375]
We propose a comprehensive approach to explore and analyze the causality of end-to-end autonomous driving.
Our work is the first to unveil the mystery of end-to-end autonomous driving and turn the black box into a white one.
arXiv Detail & Related papers (2024-07-09T04:56:11Z) - Work-in-Progress: Crash Course: Can (Under Attack) Autonomous Driving Beat Human Drivers? [60.51287814584477]
This paper evaluates the inherent risks in autonomous driving by examining the current landscape of AVs.
We develop specific claims highlighting the delicate balance between the advantages of AVs and potential security challenges in real-world scenarios.
arXiv Detail & Related papers (2024-05-14T09:42:21Z) - DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in
Autonomous Driving [65.04871316921327]
This paper introduces a new autonomous driving system that enhances the performance and reliability of autonomous driving system.
DME-Driver utilizes a powerful vision language model as the decision-maker and a planning-oriented perception model as the control signal generator.
By leveraging this dataset, our model achieves high-precision planning accuracy through a logical thinking process.
arXiv Detail & Related papers (2024-01-08T03:06:02Z) - Towards Knowledge-driven Autonomous Driving [37.003908817857095]
This paper explores the emerging knowledge-driven autonomous driving technologies.
Our investigation highlights the limitations of current autonomous driving systems.
Knowledge-driven methods with the abilities of cognition, generalization and life-long learning emerge as a promising way to overcome these challenges.
arXiv Detail & Related papers (2023-12-07T14:17:17Z) - Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems.
LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning.
We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
arXiv Detail & Related papers (2023-11-28T03:13:09Z) - Enabling High-Level Machine Reasoning with Cognitive Neuro-Symbolic
Systems [67.01132165581667]
We propose to enable high-level reasoning in AI systems by integrating cognitive architectures with external neuro-symbolic components.
We illustrate a hybrid framework centered on ACT-R and we discuss the role of generative models in recent and future applications.
arXiv Detail & Related papers (2023-11-13T21:20:17Z) - LLM4Drive: A Survey of Large Language Models for Autonomous Driving [67.843551583229]
Large language models (LLMs) have demonstrated abilities including understanding context, logical reasoning, and generating answers.
In this paper, we systematically review a research line about textitLarge Language Models for Autonomous Driving (LLM4AD).
arXiv Detail & Related papers (2023-11-02T07:23:33Z) - Receive, Reason, and React: Drive as You Say with Large Language Models
in Autonomous Vehicles [13.102404404559428]
We propose a novel framework that leverages Large Language Models (LLMs) to enhance the decision-making process in autonomous vehicles.
Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks.
We also examine real-time personalization, demonstrating how LLMs can influence driving behaviors based on verbal commands.
arXiv Detail & Related papers (2023-10-12T04:56:01Z) - DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large
Language Models [30.23228092898916]
We propose the DiLu framework, which combines a Reasoning and a Reflection module to enable the system to perform decision-making based on common-sense knowledge.
Extensive experiments prove DiLu's capability to accumulate experience and demonstrate a significant advantage in generalization ability.
To the best of our knowledge, we are the first to leverage knowledge-driven capability in decision-making for autonomous vehicles.
arXiv Detail & Related papers (2023-09-28T09:41:35Z) - Drive Like a Human: Rethinking Autonomous Driving with Large Language
Models [28.957124302293966]
We explore the potential of using a large language model (LLM) to understand the driving environment in a human-like manner.
Our experiments show that the LLM exhibits the impressive ability to reason and solve long-tailed cases.
arXiv Detail & Related papers (2023-07-14T05:18:34Z) - Parallelized and Randomized Adversarial Imitation Learning for
Safety-Critical Self-Driving Vehicles [11.463476667274051]
It is essential to consider reliable ADAS function coordination to control the driving system, safely.
This paper proposes a randomized adversarial imitation learning (RAIL) algorithm.
The proposed method is able to train the decision maker that deals with the LIDAR data and controls the autonomous driving in multi-lane complex highway environments.
arXiv Detail & Related papers (2021-12-26T23:42:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.