Criticality Metrics for Relevance Classification in Safety Evaluation of Object Detection in Automated Driving

• URL: http://arxiv.org/abs/2512.15181v1
• Date: Wed, 17 Dec 2025 08:28:53 GMT
• Title: Criticality Metrics for Relevance Classification in Safety Evaluation of Object Detection in Automated Driving
• Authors: Jörg Gamerdinger, Sven Teufel, Stephan Amann, Oliver Bringmann,
• Abstract summary: Key component for safety evaluation is the ability to distinguish between relevant and non-relevant objects.<n>This paper presents the first in-depth analysis of criticality metrics for safety evaluation of object detection systems.
• Score: 0.5701177763922466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ensuring safety is the primary objective of automated driving, which necessitates a comprehensive and accurate perception of the environment. While numerous performance evaluation metrics exist for assessing perception capabilities, incorporating safety-specific metrics is essential to reliably evaluate object detection systems. A key component for safety evaluation is the ability to distinguish between relevant and non-relevant objects - a challenge addressed by criticality or relevance metrics. This paper presents the first in-depth analysis of criticality metrics for safety evaluation of object detection systems. Through a comprehensive review of existing literature, we identify and assess a range of applicable metrics. Their effectiveness is empirically validated using the DeepAccident dataset, which features a variety of safety-critical scenarios. To enhance evaluation accuracy, we propose two novel application strategies: bidirectional criticality rating and multi-metric aggregation. Our approach demonstrates up to a 100% improvement in terms of criticality classification accuracy, highlighting its potential to significantly advance the safety evaluation of object detection systems in automated vehicles.

Related papers

• EPSM: A Novel Metric to Evaluate the Safety of Environmental Perception in Autonomous Driving [0.5314069314483559]
  It is important to evaluate not only the overall performance of perception systems, but also their safety.<n>We introduce a novel safety metric for jointly evaluating the most critical perception tasks, object and lane detection.<n>Our proposed framework integrates a new, lightweight object safety metric that quantifies the potential risk associated with object detection errors.
  arXiv  Detail & Related papers  (2025-12-17T08:46:49Z)
• A Comprehensive Safety Metric to Evaluate Perception in Autonomous Systems [39.67816845884978]
  Object perception is the main component of automotive surround sensing.<n>Various metrics already exist for the evaluation of object perception.<n>We propose a new safety metric that incorporates all these parameters and returns a single easily interpretable safety assessment score for object perception.
  arXiv  Detail & Related papers  (2025-12-16T12:53:00Z)
• Rethinking Evaluation of Infrared Small Target Detection [105.59753496831739]
  This paper introduces a hybrid-level metric incorporating pixel- and target-level performance, proposing a systematic error analysis method, and emphasizing the importance of cross-dataset evaluation.<n>An open-source toolkit has be released to facilitate standardized benchmarking.
  arXiv  Detail & Related papers  (2025-09-21T02:45:07Z)
• The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs [57.1838332916627]
  Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP)<n>Their widespread deployment has also raised significant safety concerns.<n>LLMs-generated content can exhibit unsafe behaviors such as toxicity, bias, or misinformation, especially in adversarial contexts.
  arXiv  Detail & Related papers  (2025-06-06T05:50:50Z)
• Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods [0.0]
  This literature review consolidates the rapidly evolving field of AI safety evaluations.<n>It proposes a systematic taxonomy around three dimensions: what properties we measure, how we measure them, and how these measurements integrate into frameworks.
  arXiv  Detail & Related papers  (2025-05-08T16:55:07Z)
• Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderation [52.83870601473094]
  Embodied agents exhibit immense potential across a multitude of domains.<n>Existing research predominantly concentrates on the security of general large language models.<n>This paper introduces a novel input moderation framework, meticulously designed to safeguard embodied agents.
  arXiv  Detail & Related papers  (2025-04-22T08:34:35Z)
• LSM: A Comprehensive Metric for Assessing the Safety of Lane Detection Systems in Autonomous Driving [0.5326090003728084]
  We propose the Lane Safety Metric (LSM) to evaluate the safety of lane detection systems. Additional factors such as the semantics of the scene with road type and road width should be considered for the evaluation of lane detection. We evaluate our offline safety metric on various virtual scenarios using different lane detection approaches and compare it with state-of-the-art performance metrics.
  arXiv  Detail & Related papers  (2024-07-10T15:11:37Z)
• ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models [65.79770974145983]
  ASSERT, Automated Safety Scenario Red Teaming, consists of three methods -- semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. We find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings.
  arXiv  Detail & Related papers  (2023-10-14T17:10:28Z)
• CertainNet: Sampling-free Uncertainty Estimation for Object Detection [65.28989536741658]
  Estimating the uncertainty of a neural network plays a fundamental role in safety-critical settings. In this work, we propose a novel sampling-free uncertainty estimation method for object detection. We call it CertainNet, and it is the first to provide separate uncertainties for each output signal: objectness, class, location and size.
  arXiv  Detail & Related papers  (2021-10-04T17:59:31Z)
• A Review of Testing Object-Based Environment Perception for Safe Automated Driving [0.0]
  Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance.
  arXiv  Detail & Related papers  (2021-02-16T21:40:39Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.