nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles

• URL: http://arxiv.org/abs/2106.11810v1
• Date: Tue, 22 Jun 2021 14:24:55 GMT
• Title: nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles
• Authors: Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, Sammy Omari
• Abstract summary: We propose the world's first closed-loop ML-based planning benchmark for autonomous driving. We provide a high-quality dataset with 1500h of human driving data from 4 cities across the US and Asia. We plan to release the dataset at NeurIPS 2021 and organize benchmark challenges starting in early 2022.
• Score: 7.212066200339641
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we propose the world's first closed-loop ML-based planning benchmark for autonomous driving. While there is a growing body of ML-based motion planners, the lack of established datasets and metrics has limited the progress in this area. Existing benchmarks for autonomous vehicle motion prediction have focused on short-term motion forecasting, rather than long-term planning. This has led previous works to use open-loop evaluation with L2-based metrics, which are not suitable for fairly evaluating long-term planning. Our benchmark overcomes these limitations by introducing a large-scale driving dataset, lightweight closed-loop simulator, and motion-planning-specific metrics. We provide a high-quality dataset with 1500h of human driving data from 4 cities across the US and Asia with widely varying traffic patterns (Boston, Pittsburgh, Las Vegas and Singapore). We will provide a closed-loop simulation framework with reactive agents and provide a large set of both general and scenario-specific planning metrics. We plan to release the dataset at NeurIPS 2021 and organize benchmark challenges starting in early 2022.

Related papers

• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking [65.24988062003096]
  We present NAVSIM, a framework for benchmarking vision-based driving policies. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights.
  arXiv  Detail & Related papers  (2024-06-21T17:59:02Z)
• Planning with Adaptive World Models for Autonomous Driving [50.4439896514353]
  Motion planners (MPs) are crucial for safe navigation in complex urban environments. nuPlan, a recently released MP benchmark, addresses this limitation by augmenting real-world driving logs with closed-loop simulation logic. We present AdaptiveDriver, a model-predictive control (MPC) based planner that unrolls different world models conditioned on BehaviorNet's predictions.
  arXiv  Detail & Related papers  (2024-06-15T18:53:45Z)
• Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving [59.705635382104454]
  We present Bench2Drive, the first benchmark for evaluating E2E-AD systems' multiple abilities in a closed-loop manner. We implement state-of-the-art E2E-AD models and evaluate them in Bench2Drive, providing insights regarding current status and future directions.
  arXiv  Detail & Related papers  (2024-06-06T09:12:30Z)
• Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios? [11.917542484123134]
  Real-world autonomous driving systems must make safe decisions in the face of rare and diverse traffic scenarios. Current state-of-the-art planners are mostly evaluated on real-world datasets like nuScenes (open-loop) or nuPlan (closed-loop)
  arXiv  Detail & Related papers  (2024-04-11T08:57:48Z)
• Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving [2.6855803445552233]
  nuPlan is the world's first real-world autonomous driving dataset and benchmark. The benchmark is designed to test the ability of ML-based planners to handle diverse driving situations. We present a detailed analysis of numerous baselines and investigate gaps between ML-based and traditional methods.
  arXiv  Detail & Related papers  (2024-03-07T01:24:59Z)
• TravelPlanner: A Benchmark for Real-World Planning with Language Agents [63.199454024966506]
  We propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%.
  arXiv  Detail & Related papers  (2024-02-02T18:39:51Z)
• LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning [65.86754998249224]
  We develop a novel hybrid planner that leverages a conventional rule-based planner in conjunction with an LLM-based planner. Our approach navigates complex scenarios which existing planners struggle with, produces well-reasoned outputs while also remaining grounded through working alongside the rule-based approach.
  arXiv  Detail & Related papers  (2023-12-30T02:53:45Z)
• Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? [84.17711168595311]
  End-to-end autonomous driving has emerged as a promising research direction to target autonomy from a full-stack perspective. nuScenes dataset, characterized by relatively simple driving scenarios, leads to an under-utilization of perception information in end-to-end models. We introduce a new metric to evaluate whether the predicted trajectories adhere to the road.
  arXiv  Detail & Related papers  (2023-12-05T11:32:31Z)
• Parting with Misconceptions about Learning-based Vehicle Motion Planning [30.39229175273061]
  nuPlan marks a new era in vehicle motion planning research. Existing systems struggle to simultaneously meet both requirements. We propose an extremely simple and efficient planner which outperforms an extensive set of competitors.
  arXiv  Detail & Related papers  (2023-06-13T17:57:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.