Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving

• URL: http://arxiv.org/abs/2406.03877v2
• Date: Tue, 11 Jun 2024 16:06:32 GMT
• Title: Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving
• Authors: Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, Junchi Yan,
• Abstract summary: We present Bench2Drive, the first benchmark for evaluating E2E-AD systems' multiple abilities in a closed-loop manner. We implement state-of-the-art E2E-AD models and evaluate them in Bench2Drive, providing insights regarding current status and future directions.
• Score: 59.705635382104454
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In an era marked by the rapid scaling of foundation models, autonomous driving technologies are approaching a transformative threshold where end-to-end autonomous driving (E2E-AD) emerges due to its potential of scaling up in the data-driven manner. However, existing E2E-AD methods are mostly evaluated under the open-loop log-replay manner with L2 errors and collision rate as metrics (e.g., in nuScenes), which could not fully reflect the driving performance of algorithms as recently acknowledged in the community. For those E2E-AD methods evaluated under the closed-loop protocol, they are tested in fixed routes (e.g., Town05Long and Longest6 in CARLA) with the driving score as metrics, which is known for high variance due to the unsmoothed metric function and large randomness in the long route. Besides, these methods usually collect their own data for training, which makes algorithm-level fair comparison infeasible. To fulfill the paramount need of comprehensive, realistic, and fair testing environments for Full Self-Driving (FSD), we present Bench2Drive, the first benchmark for evaluating E2E-AD systems' multiple abilities in a closed-loop manner. Bench2Drive's official training data consists of 2 million fully annotated frames, collected from 10000 short clips uniformly distributed under 44 interactive scenarios (cut-in, overtaking, detour, etc), 23 weathers (sunny, foggy, rainy, etc), and 12 towns (urban, village, university, etc) in CARLA v2. Its evaluation protocol requires E2E-AD models to pass 44 interactive scenarios under different locations and weathers which sums up to 220 routes and thus provides a comprehensive and disentangled assessment about their driving capability under different situations. We implement state-of-the-art E2E-AD models and evaluate them in Bench2Drive, providing insights regarding current status and future directions.

Related papers

• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking [65.24988062003096]
  We present NAVSIM, a framework for benchmarking vision-based driving policies. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights.
  arXiv  Detail & Related papers  (2024-06-21T17:59:02Z)
• DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving [81.04174379726251]
  This paper collects a comprehensive end-to-end driving dataset named DriveCoT. It contains sensor data, control decisions, and chain-of-thought labels to indicate the reasoning process. We propose a baseline model called DriveCoT-Agent, trained on our dataset, to generate chain-of-thought predictions and final decisions.
  arXiv  Detail & Related papers  (2024-03-25T17:59:01Z)
• Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving? [84.17711168595311]
  End-to-end autonomous driving has emerged as a promising research direction to target autonomy from a full-stack perspective. nuScenes dataset, characterized by relatively simple driving scenarios, leads to an under-utilization of perception information in end-to-end models. We introduce a new metric to evaluate whether the predicted trajectories adhere to the road.
  arXiv  Detail & Related papers  (2023-12-05T11:32:31Z)
• End-to-End Driving via Self-Supervised Imitation Learning Using Camera and LiDAR Data [6.849144123909844]
  This letter proposes the first fully self-supervised learning framework, self-supervised imitation learning (SSIL), for E2E driving. The proposed SSIL framework can learn E2E driving networks without using driving command data. Our numerical experiments with three different benchmark datasets demonstrate that the proposed SSIL framework achieves very comparable E2E driving accuracy with the supervised learning counterpart.
  arXiv  Detail & Related papers  (2023-08-28T06:17:15Z)
• 4Seasons: Benchmarking Visual SLAM and Long-Term Localization for Autonomous Driving in Challenging Conditions [54.59279160621111]
  We present a novel visual SLAM and long-term localization benchmark for autonomous driving in challenging conditions based on the large-scale 4Seasons dataset. The proposed benchmark provides drastic appearance variations caused by seasonal changes and diverse weather and illumination conditions. We introduce a new unified benchmark for jointly evaluating visual odometry, global place recognition, and map-based visual localization performance.
  arXiv  Detail & Related papers  (2022-12-31T13:52:36Z)
• Fully End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-Agent [2.512827436728378]
  We propose a novel deep learning model trained with end-to-end and multi-task learning manners to perform both perception and control tasks simultaneously. The model is evaluated on CARLA simulator with various scenarios made of normal-adversarial situations and different weathers to mimic real-world conditions.
  arXiv  Detail & Related papers  (2022-04-12T03:57:01Z)
• Towards Driving-Oriented Metric for Lane Detection Models [19.81163190104571]
  We design 2 new driving-oriented metrics for lane detection: End-to-End Lateral Deviation metric (E2E-LD) and Per-frame Simulated Lateral Deviation metric (PSLD) To evaluate the validity of the metrics, we conduct a large-scale empirical study with 4 major types of lane detection approaches on the TuSimple dataset and our newly constructed dataset Comma2k19-LD.
  arXiv  Detail & Related papers  (2022-03-31T07:24:44Z)
• Hybrid Model and Data Driven Algorithm for Online Learning of Any-to-Any Path Loss Maps [19.963385352536616]
  Learning any-to-any path loss maps might be a key enabler for applications that rely on device-to-any (D2D) communication. Model-based methods have the advantage that they can generate reliable estimations with low computational complexity. Pure data-driven methods can achieve good performance without assuming any physical model. We propose a novel hybrid model and data-driven approach that obtained datasets from an online fashion.
  arXiv  Detail & Related papers  (2021-07-14T13:08:25Z)
• SODA10M: Towards Large-Scale Object Detection Benchmark for Autonomous Driving [94.11868795445798]
  We release a Large-Scale Object Detection benchmark for Autonomous driving, named as SODA10M, containing 10 million unlabeled images and 20K images labeled with 6 representative object categories. To improve diversity, the images are collected every ten seconds per frame within 32 different cities under different weather conditions, periods and location scenes. We provide extensive experiments and deep analyses of existing supervised state-of-the-art detection models, popular self-supervised and semi-supervised approaches, and some insights about how to develop future models.
  arXiv  Detail & Related papers  (2021-06-21T13:55:57Z)
• Data Driven Prediction Architecture for Autonomous Driving and its Application on Apollo Platform [1.1142354615369274]
  We introduce a highly automated learning-based prediction model pipeline to support different prediction learning sub-modules' data annotation, feature extraction, model training/tuning and deployment. This pipeline is completely automatic without any human intervention and shows an up to 400% efficiency increase in parameter tuning, when deployed at scale in different scenarios across nations.
  arXiv  Detail & Related papers  (2020-06-11T18:16:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.