Large Language Models for Human-like Autonomous Driving: A Survey

• URL: http://arxiv.org/abs/2407.19280v1
• Date: Sat, 27 Jul 2024 15:24:11 GMT
• Title: Large Language Models for Human-like Autonomous Driving: A Survey
• Authors: Yun Li, Kai Katsumata, Ehsan Javanmardi, Manabu Tsukada,
• Abstract summary: Large Language Models (LLMs) are AI models trained on massive text corpora with remarkable language understanding and generation capabilities. This survey provides a review of progress in leveraging LLMs for Autonomous Driving. It focuses on their applications in modular AD pipelines and end-to-end AD systems.
• Score: 7.125039718268125
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs), AI models trained on massive text corpora with remarkable language understanding and generation capabilities, are transforming the field of Autonomous Driving (AD). As AD systems evolve from rule-based and optimization-based methods to learning-based techniques like deep reinforcement learning, they are now poised to embrace a third and more advanced category: knowledge-based AD empowered by LLMs. This shift promises to bring AD closer to human-like AD. However, integrating LLMs into AD systems poses challenges in real-time inference, safety assurance, and deployment costs. This survey provides a comprehensive and critical review of recent progress in leveraging LLMs for AD, focusing on their applications in modular AD pipelines and end-to-end AD systems. We highlight key advancements, identify pressing challenges, and propose promising research directions to bridge the gap between LLMs and AD, thereby facilitating the development of more human-like AD systems. The survey first introduces LLMs' key features and common training schemes, then delves into their applications in modular AD pipelines and end-to-end AD, respectively, followed by discussions on open challenges and future directions. Through this in-depth analysis, we aim to provide insights and inspiration for researchers and practitioners working at the intersection of AI and autonomous vehicles, ultimately contributing to safer, smarter, and more human-centric AD technologies.

Related papers

• Application of Multimodal Large Language Models in Autonomous Driving [1.8181868280594944]
  We conduct in-depth study on implementing the Multi-modal Large Language Model. We address problems with the poor performance of MLLM on Autonomous Driving. We then break down the AD decision-making process by scene understanding, prediction, and decision-making.
  arXiv  Detail & Related papers  (2024-12-21T00:09:52Z)
• AD-LLM: Benchmarking Large Language Models for Anomaly Detection [50.57641458208208]
  This paper introduces AD-LLM, the first benchmark that evaluates how large language models can help with anomaly detection. We examine three key tasks: zero-shot detection, using LLMs' pre-trained knowledge to perform AD without tasks-specific training; data augmentation, generating synthetic data and category descriptions to improve AD models; and model selection, using LLMs to suggest unsupervised AD models.
  arXiv  Detail & Related papers  (2024-12-15T10:22:14Z)
• A Survey on Large Language Model-empowered Autonomous Driving [25.963195890376646]
  Development of autonomous driving (AD) technology follows two main technical paths: modularization and end-to-end. This paper conducts a thorough analysis of the potential applications of large language models (LLMs) in AD systems. We discuss an important question: Can LLM-based artificial general intelligence (AGI) be a key to achieve high-level AD?
  arXiv  Detail & Related papers  (2024-09-21T15:07:37Z)
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models [71.25225058845324]
  Large Language Models (LLMs) have demonstrated revolutionary abilities in language understanding and generation. Retrieval-Augmented Generation (RAG) can offer reliable and up-to-date external knowledge. RA-LLMs have emerged to harness external and authoritative knowledge bases, rather than relying on the model's internal knowledge.
  arXiv  Detail & Related papers  (2024-05-10T02:48:45Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)
• LLM4Drive: A Survey of Large Language Models for Autonomous Driving [62.10344445241105]
  Large language models (LLMs) have demonstrated abilities including understanding context, logical reasoning, and generating answers. In this paper, we systematically review a research line about textitLarge Language Models for Autonomous Driving (LLM4AD).
  arXiv  Detail & Related papers  (2023-11-02T07:23:33Z)
• LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving [87.1164964709168]
  This work employs Large Language Models (LLMs) as a decision-making component for complex autonomous driving scenarios. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination.
  arXiv  Detail & Related papers  (2023-10-04T17:59:49Z)
• Recommender Systems in the Era of Large Language Models (LLMs) [62.0129013439038]
  Large Language Models (LLMs) have revolutionized the fields of Natural Language Processing (NLP) and Artificial Intelligence (AI) We conduct a comprehensive review of LLM-empowered recommender systems from various aspects including Pre-training, Fine-tuning, and Prompting.
  arXiv  Detail & Related papers  (2023-07-05T06:03:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.