A Comprehensive Safety Metric to Evaluate Perception in Autonomous Systems

• URL: http://arxiv.org/abs/2512.14367v1
• Date: Tue, 16 Dec 2025 12:53:00 GMT
• Title: A Comprehensive Safety Metric to Evaluate Perception in Autonomous Systems
• Authors: Georg Volk, Jörg Gamerdinger, Alexander von Bernuth, Oliver Bringmann,
• Abstract summary: Object perception is the main component of automotive surround sensing.<n>Various metrics already exist for the evaluation of object perception.<n>We propose a new safety metric that incorporates all these parameters and returns a single easily interpretable safety assessment score for object perception.
• Score: 39.67816845884978
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Complete perception of the environment and its correct interpretation is crucial for autonomous vehicles. Object perception is the main component of automotive surround sensing. Various metrics already exist for the evaluation of object perception. However, objects can be of different importance depending on their velocity, orientation, distance, size, or the potential damage that could be caused by a collision due to a missed detection. Thus, these additional parameters have to be considered for safety evaluation. We propose a new safety metric that incorporates all these parameters and returns a single easily interpretable safety assessment score for object perception. This new metric is evaluated with both real world and virtual data sets and compared to state of the art metrics.

Related papers

• EPSM: A Novel Metric to Evaluate the Safety of Environmental Perception in Autonomous Driving [0.5314069314483559]
  It is important to evaluate not only the overall performance of perception systems, but also their safety.<n>We introduce a novel safety metric for jointly evaluating the most critical perception tasks, object and lane detection.<n>Our proposed framework integrates a new, lightweight object safety metric that quantifies the potential risk associated with object detection errors.
  arXiv  Detail & Related papers  (2025-12-17T08:46:49Z)
• Criticality Metrics for Relevance Classification in Safety Evaluation of Object Detection in Automated Driving [0.5701177763922466]
  Key component for safety evaluation is the ability to distinguish between relevant and non-relevant objects.<n>This paper presents the first in-depth analysis of criticality metrics for safety evaluation of object detection systems.
  arXiv  Detail & Related papers  (2025-12-17T08:28:53Z)
• Objectness Similarity: Capturing Object-Level Fidelity in 3D Scene Evaluation [57.48883165305971]
  Existing metrics assess overall image quality, leading to discrepancies with human perception.<n>Inspired by neuropsychological insights, we hypothesize that human recognition of 3D scenes fundamentally involves attention to individual objects.<n>This paper presents Objectness SIMilarity (OSIM), a novel evaluation metric for 3D scenes that explicitly focuses on "objects"
  arXiv  Detail & Related papers  (2025-09-11T04:33:27Z)
• LSM: A Comprehensive Metric for Assessing the Safety of Lane Detection Systems in Autonomous Driving [0.5326090003728084]
  We propose the Lane Safety Metric (LSM) to evaluate the safety of lane detection systems. Additional factors such as the semantics of the scene with road type and road width should be considered for the evaluation of lane detection. We evaluate our offline safety metric on various virtual scenarios using different lane detection approaches and compare it with state-of-the-art performance metrics.
  arXiv  Detail & Related papers  (2024-07-10T15:11:37Z)
• CarPatch: A Synthetic Benchmark for Radiance Field Evaluation on Vehicle Components [77.33782775860028]
  We introduce CarPatch, a novel synthetic benchmark of vehicles. In addition to a set of images annotated with their intrinsic and extrinsic camera parameters, the corresponding depth maps and semantic segmentation masks have been generated for each view. Global and part-based metrics have been defined and used to evaluate, compare, and better characterize some state-of-the-art techniques.
  arXiv  Detail & Related papers  (2023-07-24T11:59:07Z)
• Camera-Radar Perception for Autonomous Vehicles and ADAS: Concepts, Datasets and Metrics [77.34726150561087]
  This work aims to carry out a study on the current scenario of camera and radar-based perception for ADAS and autonomous vehicles. Concepts and characteristics related to both sensors, as well as to their fusion, are presented. We give an overview of the Deep Learning-based detection and segmentation tasks, and the main datasets, metrics, challenges, and open questions in vehicle perception.
  arXiv  Detail & Related papers  (2023-03-08T00:48:32Z)
• Sensor Visibility Estimation: Metrics and Methods for Systematic Performance Evaluation and Improvement [0.0]
  We introduce metrics and a framework to assess the performance of visibility estimators. Our metrics are verified with labeled real-world and simulation data from infrastructure radars and cameras. Applying our metrics, we enhance the radar and camera visibility estimators by modeling the 3D elevation of sensor and objects.
  arXiv  Detail & Related papers  (2022-11-11T16:17:43Z)
• A Quality Index Metric and Method for Online Self-Assessment of Autonomous Vehicles Sensory Perception [164.93739293097605]
  We propose a novel evaluation metric, named as the detection quality index (DQI), which assesses the performance of camera-based object detection algorithms. We have developed a superpixel-based attention network (SPA-NET) that utilizes raw image pixels and superpixels as input to predict the proposed DQI evaluation metric.
  arXiv  Detail & Related papers  (2022-03-04T22:16:50Z)
• Evaluating Object (mis)Detection from a Safety and Reliability Perspective: Discussion and Measures [1.8492669447784602]
  We propose new object detection measures that reward the correct identification of objects that are most dangerous and most likely to affect driving decisions. We apply our model on the recent autonomous driving dataset nuScenes, and we compare nine object detectors. Results show that, in several settings, object detectors that perform best according to the nuScenes ranking are not the preferable ones when the focus is shifted on safety and reliability.
  arXiv  Detail & Related papers  (2022-03-04T09:31:20Z)
• Exploring Credibility Scoring Metrics of Perception Systems for Autonomous Driving [0.0]
  We show that offline metrics can be used to account for real-world corruptions such as poor weather conditions. This is a clear next step as it can allow for error-free autonomous vehicle perception and safer time-critical and safety-critical decision-making.
  arXiv  Detail & Related papers  (2021-12-22T03:17:14Z)
• Towards robust sensing for Autonomous Vehicles: An adversarial perspective [82.83630604517249]
  It is of primary importance that the resulting decisions are robust to perturbations. Adversarial perturbations are purposefully crafted alterations of the environment or of the sensory measurements. A careful evaluation of the vulnerabilities of their sensing system(s) is necessary in order to build and deploy safer systems.
  arXiv  Detail & Related papers  (2020-07-14T05:25:15Z)
• Learning to Evaluate Perception Models Using Planner-Centric Metrics [104.33349410009161]
  We propose a principled metric for 3D object detection specifically for the task of self-driving. We find that our metric penalizes many of the mistakes that other metrics penalize by design. For human evaluation, we generate scenes in which standard metrics and our metric disagree and find that humans side with our metric 79% of the time.
  arXiv  Detail & Related papers  (2020-04-19T02:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.