A Language Agent for Autonomous Driving

• URL: http://arxiv.org/abs/2311.10813v4
• Date: Sun, 28 Jul 2024 23:37:51 GMT
• Title: A Language Agent for Autonomous Driving
• Authors: Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, Yue Wang,
• Abstract summary: We propose a paradigm shift to integrate human-like intelligence into autonomous driving systems. Our approach, termed Agent-Driver, transforms the traditional autonomous driving pipeline by introducing a versatile tool library. Powered by Large Language Models (LLMs), our Agent-Driver is endowed with intuitive common sense and robust reasoning capabilities.
• Score: 31.359413767191608
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Human-level driving is an ultimate goal of autonomous driving. Conventional approaches formulate autonomous driving as a perception-prediction-planning framework, yet their systems do not capitalize on the inherent reasoning ability and experiential knowledge of humans. In this paper, we propose a fundamental paradigm shift from current pipelines, exploiting Large Language Models (LLMs) as a cognitive agent to integrate human-like intelligence into autonomous driving systems. Our approach, termed Agent-Driver, transforms the traditional autonomous driving pipeline by introducing a versatile tool library accessible via function calls, a cognitive memory of common sense and experiential knowledge for decision-making, and a reasoning engine capable of chain-of-thought reasoning, task planning, motion planning, and self-reflection. Powered by LLMs, our Agent-Driver is endowed with intuitive common sense and robust reasoning capabilities, thus enabling a more nuanced, human-like approach to autonomous driving. We evaluate our approach on the large-scale nuScenes benchmark, and extensive experiments substantiate that our Agent-Driver significantly outperforms the state-of-the-art driving methods by a large margin. Our approach also demonstrates superior interpretability and few-shot learning ability to these methods.

Related papers

• Exploring the Causality of End-to-End Autonomous Driving [57.631400236930375]
  We propose a comprehensive approach to explore and analyze the causality of end-to-end autonomous driving. Our work is the first to unveil the mystery of end-to-end autonomous driving and turn the black box into a white one.
  arXiv  Detail & Related papers  (2024-07-09T04:56:11Z)
• Work-in-Progress: Crash Course: Can (Under Attack) Autonomous Driving Beat Human Drivers? [60.51287814584477]
  This paper evaluates the inherent risks in autonomous driving by examining the current landscape of AVs. We develop specific claims highlighting the delicate balance between the advantages of AVs and potential security challenges in real-world scenarios.
  arXiv  Detail & Related papers  (2024-05-14T09:42:21Z)
• DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving [65.04871316921327]
  This paper introduces a new autonomous driving system that enhances the performance and reliability of autonomous driving system. DME-Driver utilizes a powerful vision language model as the decision-maker and a planning-oriented perception model as the control signal generator. By leveraging this dataset, our model achieves high-precision planning accuracy through a logical thinking process.
  arXiv  Detail & Related papers  (2024-01-08T03:06:02Z)
• Towards Knowledge-driven Autonomous Driving [37.003908817857095]
  This paper explores the emerging knowledge-driven autonomous driving technologies. Our investigation highlights the limitations of current autonomous driving systems. Knowledge-driven methods with the abilities of cognition, generalization and life-long learning emerge as a promising way to overcome these challenges.
  arXiv  Detail & Related papers  (2023-12-07T14:17:17Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)
• Enabling High-Level Machine Reasoning with Cognitive Neuro-Symbolic Systems [67.01132165581667]
  We propose to enable high-level reasoning in AI systems by integrating cognitive architectures with external neuro-symbolic components. We illustrate a hybrid framework centered on ACT-R and we discuss the role of generative models in recent and future applications.
  arXiv  Detail & Related papers  (2023-11-13T21:20:17Z)
• LLM4Drive: A Survey of Large Language Models for Autonomous Driving [62.10344445241105]
  Large language models (LLMs) have demonstrated abilities including understanding context, logical reasoning, and generating answers. In this paper, we systematically review a research line about textitLarge Language Models for Autonomous Driving (LLM4AD).
  arXiv  Detail & Related papers  (2023-11-02T07:23:33Z)
• Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles [13.102404404559428]
  We propose a novel framework that leverages Large Language Models (LLMs) to enhance the decision-making process in autonomous vehicles. Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks. We also examine real-time personalization, demonstrating how LLMs can influence driving behaviors based on verbal commands.
  arXiv  Detail & Related papers  (2023-10-12T04:56:01Z)
• DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models [30.23228092898916]
  We propose the DiLu framework, which combines a Reasoning and a Reflection module to enable the system to perform decision-making based on common-sense knowledge. Extensive experiments prove DiLu's capability to accumulate experience and demonstrate a significant advantage in generalization ability. To the best of our knowledge, we are the first to leverage knowledge-driven capability in decision-making for autonomous vehicles.
  arXiv  Detail & Related papers  (2023-09-28T09:41:35Z)
• Drive Like a Human: Rethinking Autonomous Driving with Large Language Models [28.957124302293966]
  We explore the potential of using a large language model (LLM) to understand the driving environment in a human-like manner. Our experiments show that the LLM exhibits the impressive ability to reason and solve long-tailed cases.
  arXiv  Detail & Related papers  (2023-07-14T05:18:34Z)
• Parallelized and Randomized Adversarial Imitation Learning for Safety-Critical Self-Driving Vehicles [11.463476667274051]
  It is essential to consider reliable ADAS function coordination to control the driving system, safely. This paper proposes a randomized adversarial imitation learning (RAIL) algorithm. The proposed method is able to train the decision maker that deals with the LIDAR data and controls the autonomous driving in multi-lane complex highway environments.
  arXiv  Detail & Related papers  (2021-12-26T23:42:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.