Personalized Autonomous Driving with Large Language Models: Field Experiments

• URL: http://arxiv.org/abs/2312.09397v3
• Date: Wed, 8 May 2024 17:24:33 GMT
• Title: Personalized Autonomous Driving with Large Language Models: Field Experiments
• Authors: Can Cui, Zichong Yang, Yupeng Zhou, Yunsheng Ma, Juanwu Lu, Lingxi Li, Yaobin Chen, Jitesh Panchal, Ziran Wang,
• Abstract summary: We introduce an LLM-based framework, Talk2Drive, capable of translating natural verbal commands into executable controls. This is the first-of-its-kind multi-scenario field experiment that deploys LLMs on a real-world autonomous vehicle. We validate that the proposed memory module considers personalized preferences and further reduces the takeover rate by up to 65.2%.
• Score: 11.429053835807697
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Integrating large language models (LLMs) in autonomous vehicles enables conversation with AI systems to drive the vehicle. However, it also emphasizes the requirement for such systems to comprehend commands accurately and achieve higher-level personalization to adapt to the preferences of drivers or passengers over a more extended period. In this paper, we introduce an LLM-based framework, Talk2Drive, capable of translating natural verbal commands into executable controls and learning to satisfy personal preferences for safety, efficiency, and comfort with a proposed memory module. This is the first-of-its-kind multi-scenario field experiment that deploys LLMs on a real-world autonomous vehicle. Experiments showcase that the proposed system can comprehend human intentions at different intuition levels, ranging from direct commands like "can you drive faster" to indirect commands like "I am really in a hurry now". Additionally, we use the takeover rate to quantify the trust of human drivers in the LLM-based autonomous driving system, where Talk2Drive significantly reduces the takeover rate in highway, intersection, and parking scenarios. We also validate that the proposed memory module considers personalized preferences and further reduces the takeover rate by up to 65.2% compared with those without a memory module. The experiment video can be watched at https://www.youtube.com/watch?v=4BWsfPaq1Ro

Related papers

• Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving [14.64475022650084]
  Large Language Models (LLMs) have garnered significant attention for their ability to understand text and images, generate human-like text, and perform complex reasoning tasks. We investigate how well LLMs can adapt and apply a combination of arithmetic and common-sense reasoning, particularly in autonomous driving scenarios.
  arXiv  Detail & Related papers  (2024-02-21T08:09:05Z)
• DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving [69.82743399946371]
  DriveMLM is a framework that can perform close-loop autonomous driving in realistic simulators. We employ a multi-modal LLM (MLLM) to model the behavior planning module of a module AD system. This model can plug-and-play in existing AD systems such as Apollo for close-loop driving.
  arXiv  Detail & Related papers  (2023-12-14T18:59:05Z)
• LMDrive: Closed-Loop End-to-End Driving with Large Language Models [37.910449013471656]
  Large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence" This paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework.
  arXiv  Detail & Related papers  (2023-12-12T18:24:15Z)
• LLM4Drive: A Survey of Large Language Models for Autonomous Driving [62.10344445241105]
  Large language models (LLMs) have demonstrated abilities including understanding context, logical reasoning, and generating answers. In this paper, we systematically review a research line about textitLarge Language Models for Autonomous Driving (LLM4AD).
  arXiv  Detail & Related papers  (2023-11-02T07:23:33Z)
• Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models [114.69732301904419]
  We present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. Our approach demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations.
  arXiv  Detail & Related papers  (2023-10-26T17:56:35Z)
• Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles [13.102404404559428]
  We propose a novel framework that leverages Large Language Models (LLMs) to enhance the decision-making process in autonomous vehicles. Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks. We also examine real-time personalization, demonstrating how LLMs can influence driving behaviors based on verbal commands.
  arXiv  Detail & Related papers  (2023-10-12T04:56:01Z)
• LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving [87.1164964709168]
  This work employs Large Language Models (LLMs) as a decision-making component for complex autonomous driving scenarios. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination.
  arXiv  Detail & Related papers  (2023-10-04T17:59:49Z)
• Tackling Real-World Autonomous Driving using Deep Reinforcement Learning [63.3756530844707]
  In this work, we propose a model-free Deep Reinforcement Learning Planner training a neural network that predicts acceleration and steering angle. In order to deploy the system on board the real self-driving car, we also develop a module represented by a tiny neural network.
  arXiv  Detail & Related papers  (2022-07-05T16:33:20Z)
• An Intelligent Self-driving Truck System For Highway Transportation [81.12838700312308]
  In this paper, we introduce an intelligent self-driving truck system. Our presented system consists of three main components, 1) a realistic traffic simulation module for generating realistic traffic flow in testing scenarios, 2) a high-fidelity truck model which is designed and evaluated for mimicking real truck response in real-world deployment. We also deploy our proposed system on a real truck and conduct real world experiments which shows our system's capacity of mitigating sim-to-real gap.
  arXiv  Detail & Related papers  (2021-12-31T04:54:13Z)
• A LiDAR Assisted Control Module with High Precision in Parking Scenarios for Autonomous Driving Vehicle [39.42619778086731]
  We introduce a real-world, industrial scenario of which human drivers are not capable. A precise (3? = 2 centimeters) Error Feedback System was first built to partly replace the localization module. We show that the results not only outperformed original Apollo modules but also beat specially trained and highly experienced human test drivers.
  arXiv  Detail & Related papers  (2021-05-02T06:13:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.