Towards Driving-Oriented Metric for Lane Detection Models

• URL: http://arxiv.org/abs/2203.16851v1
• Date: Thu, 31 Mar 2022 07:24:44 GMT
• Title: Towards Driving-Oriented Metric for Lane Detection Models
• Authors: Takami Sato and Qi Alfred Chen
• Abstract summary: We design 2 new driving-oriented metrics for lane detection: End-to-End Lateral Deviation metric (E2E-LD) and Per-frame Simulated Lateral Deviation metric (PSLD) To evaluate the validity of the metrics, we conduct a large-scale empirical study with 4 major types of lane detection approaches on the TuSimple dataset and our newly constructed dataset Comma2k19-LD.
• Score: 19.81163190104571
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: After the 2017 TuSimple Lane Detection Challenge, its dataset and evaluation based on accuracy and F1 score have become the de facto standard to measure the performance of lane detection methods. While they have played a major role in improving the performance of lane detection methods, the validity of this evaluation method in downstream tasks has not been adequately researched. In this study, we design 2 new driving-oriented metrics for lane detection: End-to-End Lateral Deviation metric (E2E-LD) is directly formulated based on the requirements of autonomous driving, a core downstream task of lane detection; Per-frame Simulated Lateral Deviation metric (PSLD) is a lightweight surrogate metric of E2E-LD. To evaluate the validity of the metrics, we conduct a large-scale empirical study with 4 major types of lane detection approaches on the TuSimple dataset and our newly constructed dataset Comma2k19-LD. Our results show that the conventional metrics have strongly negative correlations ($\leq$-0.55) with E2E-LD, meaning that some recent improvements purely targeting the conventional metrics may not have led to meaningful improvements in autonomous driving, but rather may actually have made it worse by overfitting to the conventional metrics. As autonomous driving is a security/safety-critical system, the underestimation of robustness hinders the sound development of practical lane detection models. We hope that our study will help the community achieve more downstream task-aware evaluations for lane detection.

Related papers

• Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving [59.705635382104454]
  We present Bench2Drive, the first benchmark for evaluating E2E-AD systems' multiple abilities in a closed-loop manner. We implement state-of-the-art E2E-AD models and evaluate them in Bench2Drive, providing insights regarding current status and future directions.
  arXiv  Detail & Related papers  (2024-06-06T09:12:30Z)
• DRUformer: Enhancing the driving scene Important object detection with driving relationship self-understanding [50.81809690183755]
  Traffic accidents frequently lead to fatal injuries, contributing to over 50 million deaths until 2023. Previous research primarily assessed the importance of individual participants, treating them as independent entities. We introduce Driving scene Relationship self-Understanding transformer (DRUformer) to enhance the important object detection task.
  arXiv  Detail & Related papers  (2023-11-11T07:26:47Z)
• Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features [69.47588461101925]
  We propose a method to adapt 3D object detectors to new driving environments. Our approach enhances LiDAR-based detection models using spatial quantized historical features. Experiments on real-world datasets demonstrate significant improvements.
  arXiv  Detail & Related papers  (2023-09-21T15:00:31Z)
• Decoupling the Curve Modeling and Pavement Regression for Lane Detection [67.22629246312283]
  curve-based lane representation is a popular approach in many lane detection methods. We propose a new approach to the lane detection task by decomposing it into two parts: curve modeling and ground height regression.
  arXiv  Detail & Related papers  (2023-09-19T11:24:14Z)
• On Offline Evaluation of 3D Object Detection for Autonomous Driving [33.16617625256519]
  We measure how predictive different detection metrics are of driving performance when detectors are integrated into a full self-driving stack. We find that the nuScenes Detection Score has a higher correlation to driving performance than the widely used average precision metric.
  arXiv  Detail & Related papers  (2023-08-24T13:31:51Z)
• Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes [38.43491956142818]
  Planning task involves predicting the trajectory of the ego vehicle based on inputs from both internal intention and the external environment. Most existing works evaluate their performance on the nuScenes dataset using the L2 error and collision rate between the predicted trajectories and the ground truth. In this paper, we reevaluate these existing evaluation metrics and explore whether they accurately measure the superiority of different methods. Our simple method achieves similar end-to-end planning performance on the nuScenes dataset with other perception-based methods, reducing the average L2 error by about 20%.
  arXiv  Detail & Related papers  (2023-05-17T17:59:11Z)
• RONELDv2: A faster, improved lane tracking method [1.3965477771846408]
  Lane detection is an integral part of control systems in autonomous vehicles and lane departure warning systems. This paper proposes an improved, lighter weight lane detection method, RONELDv2. Experiments using the proposed improvements show a consistent increase in lane detection accuracy results across different datasets and deep learning models.
  arXiv  Detail & Related papers  (2022-02-26T13:12:09Z)
• On Robustness of Lane Detection Models to Physical-World Adversarial Attacks in Autonomous Driving [12.412448947321828]
  After the 2017 TuSimple Lane Detection Challenge, its evaluation based on accuracy and F1 score has become the de facto standard to measure the performance of lane detection methods. We conduct the first large-scale empirical study to evaluate the robustness of state-of-the-art lane detection methods under physical-world adversarial attacks in autonomous driving.
  arXiv  Detail & Related papers  (2021-07-06T09:04:47Z)
• Efficient and Robust LiDAR-Based End-to-End Navigation [132.52661670308606]
  We present an efficient and robust LiDAR-based end-to-end navigation framework. We propose Fast-LiDARNet that is based on sparse convolution kernel optimization and hardware-aware model design. We then propose Hybrid Evidential Fusion that directly estimates the uncertainty of the prediction from only a single forward pass.
  arXiv  Detail & Related papers  (2021-05-20T17:52:37Z)
• Divide-and-Conquer for Lane-Aware Diverse Trajectory Prediction [71.97877759413272]
  Trajectory prediction is a safety-critical tool for autonomous vehicles to plan and execute actions. Recent methods have achieved strong performances using Multi-Choice Learning objectives like winner-takes-all (WTA) or best-of-many. Our work addresses two key challenges in trajectory prediction, learning outputs, and better predictions by imposing constraints using driving knowledge.
  arXiv  Detail & Related papers  (2021-04-16T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.