Leveraging Large Language Models for Enhancing Autonomous Vehicle Perception

• URL: http://arxiv.org/abs/2412.20230v1
• Date: Sat, 28 Dec 2024 17:58:44 GMT
• Title: Leveraging Large Language Models for Enhancing Autonomous Vehicle Perception
• Authors: Athanasios Karagounis,
• Abstract summary: Large Language Models (LLMs) are used to address challenges in dynamic environments, sensor fusion, and contextual reasoning. This paper presents a novel framework for incorporating LLMs into AV perception, enabling advanced contextual understanding. Experimental results demonstrate that LLMs significantly improve the accuracy and reliability of AV perception systems.
• Score: 0.0
• License:
• Abstract: Autonomous vehicles (AVs) rely on sophisticated perception systems to interpret their surroundings, a cornerstone for safe navigation and decision-making. The integration of Large Language Models (LLMs) into AV perception frameworks offers an innovative approach to address challenges in dynamic environments, sensor fusion, and contextual reasoning. This paper presents a novel framework for incorporating LLMs into AV perception, enabling advanced contextual understanding, seamless sensor integration, and enhanced decision support. Experimental results demonstrate that LLMs significantly improve the accuracy and reliability of AV perception systems, paving the way for safer and more intelligent autonomous driving technologies. By expanding the scope of perception beyond traditional methods, LLMs contribute to creating a more adaptive and human-centric driving ecosystem, making autonomous vehicles more reliable and transparent in their operations. These advancements redefine the relationship between human drivers and autonomous systems, fostering trust through enhanced understanding and personalized decision-making. Furthermore, by integrating memory modules and adaptive learning mechanisms, LLMs introduce continuous improvement in AV perception, enabling vehicles to evolve with time and adapt to changing environments and user preferences.

Related papers

• Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives [56.528835143531694]
  We introduce DriveBench, a benchmark dataset designed to evaluate Vision-Language Models (VLMs) Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding. We propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding.
  arXiv  Detail & Related papers  (2025-01-07T18:59:55Z)
• SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Driving [10.041702058108482]
  This study addresses the critical need for enhanced situational awareness in autonomous driving (AD) by leveraging the contextual reasoning capabilities of large language models (LLMs) Unlike traditional perception systems that rely on rigid, label-based annotations, it integrates real-time, multimodal sensor data into a unified, LLMs-readable knowledge base. Experimental results using real-world Vehicle-to-everything (V2X) datasets demonstrate significant improvements in perception and prediction performance.
  arXiv  Detail & Related papers  (2025-01-07T05:15:46Z)
• Integrating Large Language Models for UAV Control in Simulated Environments: A Modular Interaction Approach [0.3495246564946556]
  This study explores the application of Large Language Models in UAV control. By enabling UAVs to interpret and respond to natural language commands, LLMs simplify the UAV control and usage. The paper discusses several key areas where LLMs can impact UAV technology, including autonomous decision-making, dynamic mission planning, enhanced situational awareness, and improved safety protocols.
  arXiv  Detail & Related papers  (2024-10-23T06:56:53Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)
• LLM4Drive: A Survey of Large Language Models for Autonomous Driving [62.10344445241105]
  Large language models (LLMs) have demonstrated abilities including understanding context, logical reasoning, and generating answers. In this paper, we systematically review a research line about textitLarge Language Models for Autonomous Driving (LLM4AD).
  arXiv  Detail & Related papers  (2023-11-02T07:23:33Z)
• Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models [114.69732301904419]
  We present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. Our approach demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations.
  arXiv  Detail & Related papers  (2023-10-26T17:56:35Z)
• Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles [13.102404404559428]
  We propose a novel framework that leverages Large Language Models (LLMs) to enhance the decision-making process in autonomous vehicles. Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks. We also examine real-time personalization, demonstrating how LLMs can influence driving behaviors based on verbal commands.
  arXiv  Detail & Related papers  (2023-10-12T04:56:01Z)
• LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving [87.1164964709168]
  This work employs Large Language Models (LLMs) as a decision-making component for complex autonomous driving scenarios. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination.
  arXiv  Detail & Related papers  (2023-10-04T17:59:49Z)
• Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles [13.102404404559428]
  We present a novel framework that leverages Large Language Models (LLMs) to enhance autonomous vehicles' decision-making processes. The proposed framework holds the potential to revolutionize the way autonomous vehicles operate, offering personalized assistance, continuous learning, and transparent decision-making.
  arXiv  Detail & Related papers  (2023-09-19T00:47:13Z)
• Transferable Deep Reinforcement Learning Framework for Autonomous Vehicles with Joint Radar-Data Communications [69.24726496448713]
  We propose an intelligent optimization framework based on the Markov Decision Process (MDP) to help the AV make optimal decisions. We then develop an effective learning algorithm leveraging recent advances of deep reinforcement learning techniques to find the optimal policy for the AV. We show that the proposed transferable deep reinforcement learning framework reduces the obstacle miss detection probability by the AV up to 67% compared to other conventional deep reinforcement learning approaches.
  arXiv  Detail & Related papers  (2021-05-28T08:45:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.