LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning

• URL: http://arxiv.org/abs/2401.00125v1
• Date: Sat, 30 Dec 2023 02:53:45 GMT
• Title: LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning
• Authors: S P Sharan, Francesco Pittaluga, Vijay Kumar B G, Manmohan Chandraker
• Abstract summary: We develop a novel hybrid planner that leverages a conventional rule-based planner in conjunction with an LLM-based planner. Our approach navigates complex scenarios which existing planners struggle with, produces well-reasoned outputs while also remaining grounded through working alongside the rule-based approach.
• Score: 65.86754998249224
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although planning is a crucial component of the autonomous driving stack, researchers have yet to develop robust planning algorithms that are capable of safely handling the diverse range of possible driving scenarios. Learning-based planners suffer from overfitting and poor long-tail performance. On the other hand, rule-based planners generalize well, but might fail to handle scenarios that require complex driving maneuvers. To address these limitations, we investigate the possibility of leveraging the common-sense reasoning capabilities of Large Language Models (LLMs) such as GPT4 and Llama2 to generate plans for self-driving vehicles. In particular, we develop a novel hybrid planner that leverages a conventional rule-based planner in conjunction with an LLM-based planner. Guided by commonsense reasoning abilities of LLMs, our approach navigates complex scenarios which existing planners struggle with, produces well-reasoned outputs while also remaining grounded through working alongside the rule-based approach. Through extensive evaluation on the nuPlan benchmark, we achieve state-of-the-art performance, outperforming all existing pure learning- and rule-based methods across most metrics. Our code will be available at https://llmassist.github.io.

Related papers

• Diffusion-Based Planning for Autonomous Driving with Flexible Guidance [19.204115959760788]
  We propose a novel transformer-based Diffusion Planner for closed-loop planning. Our model supports joint modeling of both prediction and planning tasks. It achieves state-of-the-art closed-loop performance with robust transferability in diverse driving styles.
  arXiv  Detail & Related papers  (2025-01-26T15:49:50Z)
• LLMs Can Plan Only If We Tell Them [16.593590353705697]
  Large language models (LLMs) have demonstrated significant capabilities in natural language processing and reasoning. This paper investigates whether LLMs can independently generate long-horizon plans that rival human baselines.
  arXiv  Detail & Related papers  (2025-01-23T10:46:14Z)
• Distilling Multi-modal Large Language Models for Autonomous Driving [64.63127269187814]
  Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events. We propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios.
  arXiv  Detail & Related papers  (2025-01-16T18:59:53Z)
• Asynchronous Large Language Model Enhanced Planner for Autonomous Driving [26.72215912937613]
  AsyncDriver is a new framework designed to guide real-time planners in making precise and controllable trajectory predictions. By capitalizing on the asynchronous nature of their inference frequencies, our approach have successfully reduced the computational cost introduced by LLM. Experiments show that our approach achieves superior closed-loop evaluation performance on nuPlan's challenging scenarios.
  arXiv  Detail & Related papers  (2024-06-20T17:59:03Z)
• Exploring and Benchmarking the Planning Capabilities of Large Language Models [57.23454975238014]
  This work lays the foundations for improving planning capabilities of large language models (LLMs) We construct a comprehensive benchmark suite encompassing both classical planning benchmarks and natural language scenarios. We investigate the use of many-shot in-context learning to enhance LLM planning, exploring the relationship between increased context length and improved planning performance.
  arXiv  Detail & Related papers  (2024-06-18T22:57:06Z)
• Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios? [11.917542484123134]
  Real-world autonomous driving systems must make safe decisions in the face of rare and diverse traffic scenarios. Current state-of-the-art planners are mostly evaluated on real-world datasets like nuScenes (open-loop) or nuPlan (closed-loop)
  arXiv  Detail & Related papers  (2024-04-11T08:57:48Z)
• LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving [87.1164964709168]
  This work employs Large Language Models (LLMs) as a decision-making component for complex autonomous driving scenarios. Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination.
  arXiv  Detail & Related papers  (2023-10-04T17:59:49Z)
• GPT-Driver: Learning to Drive with GPT [47.14350537515685]
  We present a simple yet effective approach that can transform the OpenAI GPT-3.5 model into a reliable motion planner for autonomous vehicles. We capitalize on the strong reasoning capabilities and generalization potential inherent to Large Language Models (LLMs) We evaluate our approach on the large-scale nuScenes dataset, and extensive experiments substantiate the effectiveness, generalization ability, and interpretability of our GPT-based motion planner.
  arXiv  Detail & Related papers  (2023-10-02T17:59:57Z)
• AdaPlanner: Adaptive Planning from Feedback with Language Models [56.367020818139665]
  Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities.
  arXiv  Detail & Related papers  (2023-05-26T05:52:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.