A Superalignment Framework in Autonomous Driving with Large Language Models

• URL: http://arxiv.org/abs/2406.05651v1
• Date: Sun, 9 Jun 2024 05:26:38 GMT
• Title: A Superalignment Framework in Autonomous Driving with Large Language Models
• Authors: Xiangrui Kong, Thomas Braunl, Marco Fahmi, Yue Wang,
• Abstract summary: Large language models (LLMs) and multi-modal large language models (MLLMs) are extensively used in autonomous driving. Despite their importance, the security aspect of LLMs in autonomous driving remains underexplored. This research introduces a novel security framework for autonomous vehicles, utilizing a multi-agent LLM approach.
• Score: 2.650382010271
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Over the last year, significant advancements have been made in the realms of large language models (LLMs) and multi-modal large language models (MLLMs), particularly in their application to autonomous driving. These models have showcased remarkable abilities in processing and interacting with complex information. In autonomous driving, LLMs and MLLMs are extensively used, requiring access to sensitive vehicle data such as precise locations, images, and road conditions. These data are transmitted to an LLM-based inference cloud for advanced analysis. However, concerns arise regarding data security, as the protection against data and privacy breaches primarily depends on the LLM's inherent security measures, without additional scrutiny or evaluation of the LLM's inference outputs. Despite its importance, the security aspect of LLMs in autonomous driving remains underexplored. Addressing this gap, our research introduces a novel security framework for autonomous vehicles, utilizing a multi-agent LLM approach. This framework is designed to safeguard sensitive information associated with autonomous vehicles from potential leaks, while also ensuring that LLM outputs adhere to driving regulations and align with human values. It includes mechanisms to filter out irrelevant queries and verify the safety and reliability of LLM outputs. Utilizing this framework, we evaluated the security, privacy, and cost aspects of eleven large language model-driven autonomous driving cues. Additionally, we performed QA tests on these driving prompts, which successfully demonstrated the framework's efficacy.

Related papers

• DriveMLLM: A Benchmark for Spatial Understanding with Multimodal Large Language Models in Autonomous Driving [13.115027801151484]
  We introduce DriveMLLM, a benchmark designed to evaluate the spatial understanding capabilities of multimodal large language models (MLLMs) in autonomous driving. DriveMLLM includes 880 front-facing camera images and introduces both absolute and relative spatial reasoning tasks, accompanied by linguistically diverse natural language questions. We evaluate several state-of-the-art MLLMs on DriveMLLM, and our results reveal the limitations of current models in understanding complex spatial relationships in driving contexts.
  arXiv  Detail & Related papers  (2024-11-20T08:14:01Z)
• LLM-PBE: Assessing Data Privacy in Large Language Models [111.58198436835036]
  Large Language Models (LLMs) have become integral to numerous domains, significantly advancing applications in data management, mining, and analysis. Despite the critical nature of this issue, there has been no existing literature to offer a comprehensive assessment of data privacy risks in LLMs. Our paper introduces LLM-PBE, a toolkit crafted specifically for the systematic evaluation of data privacy risks in LLMs.
  arXiv  Detail & Related papers  (2024-08-23T01:37:29Z)
• Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events [5.233512464561313]
  Multimodal Large Language Models (MLLMs) offer a novel approach by integrating textual, visual, and audio modalities. Our framework leverages the reasoning power of MLLMs, directing their output through context-specific prompts. Preliminary results demonstrate the framework's potential in zero-shot learning and accurate scenario analysis.
  arXiv  Detail & Related papers  (2024-06-19T23:50:41Z)
• AutoGuide: Automated Generation and Selection of State-Aware Guidelines for Large Language Model Agents [74.17623527375241]
  AutoGuide bridges the knowledge gap in pre-trained LLMs by leveraging implicit knowledge in offline experiences. We show that our approach outperforms competitive LLM-based baselines by a large margin in sequential decision-making benchmarks.
  arXiv  Detail & Related papers  (2024-03-13T22:06:03Z)
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension [63.330262740414646]
  We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs) We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
  arXiv  Detail & Related papers  (2024-02-28T04:56:21Z)
• ChatSOS: LLM-based knowledge Q&A system for safety engineering [0.0]
  This study introduces an LLM-based Q&A system for safety engineering, enhancing the comprehension and response accuracy of the model. We employ prompt engineering to incorporate external knowledge databases, thus enriching the LLM with up-to-date and reliable information. Our findings indicate that the integration of external knowledge significantly augments the capabilities of LLM for in-depth problem analysis and autonomous task assignment.
  arXiv  Detail & Related papers  (2023-12-14T03:25:23Z)
• Evaluation of Large Language Models for Decision Making in Autonomous Driving [4.271294502084542]
  One strategy of using Large Language Models (LLMs) for autonomous driving involves inputting surrounding objects as text prompts to the LLMs. When using LLMs for such purposes, capabilities such as spatial recognition and planning are essential. This study quantitatively evaluated these two abilities of LLMs in the context of autonomous driving.
  arXiv  Detail & Related papers  (2023-12-11T12:56:40Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)
• LLM4Drive: A Survey of Large Language Models for Autonomous Driving [62.10344445241105]
  Large language models (LLMs) have demonstrated abilities including understanding context, logical reasoning, and generating answers. In this paper, we systematically review a research line about textitLarge Language Models for Autonomous Driving (LLM4AD).
  arXiv  Detail & Related papers  (2023-11-02T07:23:33Z)
• LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving [87.1164964709168]
  This work employs Large Language Models (LLMs) as a decision-making component for complex autonomous driving scenarios. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination.
  arXiv  Detail & Related papers  (2023-10-04T17:59:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.