A Review of Testing Object-Based Environment Perception for Safe Automated Driving

• URL: http://arxiv.org/abs/2102.08460v1
• Date: Tue, 16 Feb 2021 21:40:39 GMT
• Title: A Review of Testing Object-Based Environment Perception for Safe Automated Driving
• Authors: Michael Hoss, Maike Scholtes, Lutz Eckstein
• Abstract summary: Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance. We focus on testing for verification and validation purposes at the interface between perception and planning, and structure our analysis along the three axes 1) test criteria and metrics, 2) test scenarios, and 3) reference data. Furthermore, the analyzed literature includes related safety standards, safety-independent perception algorithm benchmarking, and sensor modeling. We find that the realization of safety-aware perception testing remains an open issue since challenges concerning the three testing axes and their interdependencies currently do not appear to be sufficiently solved.

Related papers

• EPSM: A Novel Metric to Evaluate the Safety of Environmental Perception in Autonomous Driving [0.5314069314483559]
  It is important to evaluate not only the overall performance of perception systems, but also their safety.<n>We introduce a novel safety metric for jointly evaluating the most critical perception tasks, object and lane detection.<n>Our proposed framework integrates a new, lightweight object safety metric that quantifies the potential risk associated with object detection errors.
  arXiv  Detail & Related papers  (2025-12-17T08:46:49Z)
• Criticality Metrics for Relevance Classification in Safety Evaluation of Object Detection in Automated Driving [0.5701177763922466]
  Key component for safety evaluation is the ability to distinguish between relevant and non-relevant objects.<n>This paper presents the first in-depth analysis of criticality metrics for safety evaluation of object detection systems.
  arXiv  Detail & Related papers  (2025-12-17T08:28:53Z)
• CARE: Decoding Time Safety Alignment via Rollback and Introspection Intervention [68.95008546581339]
  Existing decoding-time interventions, such as Contrastive Decoding, often force a severe trade-off between safety and response quality.<n>We propose CARE, a novel framework for decoding-time safety alignment that integrates three key components.<n>The framework achieves a superior balance of safety, quality, and efficiency, attaining a low harmful response rate and minimal disruption to the user experience.
  arXiv  Detail & Related papers  (2025-09-01T04:50:02Z)
• Behavioral Safety Assessment towards Large-scale Deployment of Autonomous Vehicles [6.846750893175613]
  We propose a paradigm shift toward behavioral safety for autonomous vehicles (AVs)<n>We introduce a third-party AV safety assessment framework comprising two complementary evaluation components: Driver Licensing Test and Driving Intelligence Test.<n>We validated our proposed framework using textttAutoware.Universe, an open-source Level 4 AV, tested both in simulated environments and on the physical test track at the University of Michigan's Mcity Testing Facility.
  arXiv  Detail & Related papers  (2025-05-22T04:28:59Z)
• On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles [4.342427756164555]
  This paper argues that a rigorous statistical foundation is essential to address these challenges and enable rigorous safety assurance.<n>By drawing parallels between AV testing and established software testing methods, we identify shared research gaps and reusable solutions.<n>Our analysis reveals that neither scenario-based nor mile-based testing universally outperforms the other.
  arXiv  Detail & Related papers  (2025-05-04T22:06:23Z)
• Towards Trustworthy GUI Agents: A Survey [64.6445117343499]
  This survey examines the trustworthiness of GUI agents in five critical dimensions. We identify major challenges such as vulnerability to adversarial attacks, cascading failure modes in sequential decision-making. As GUI agents become more widespread, establishing robust safety standards and responsible development practices is essential.
  arXiv  Detail & Related papers  (2025-03-30T13:26:00Z)
• Uncertainty Estimation for 3D Object Detection via Evidential Learning [63.61283174146648]
  We introduce a framework for quantifying uncertainty in 3D object detection by leveraging an evidential learning loss on Bird's Eye View representations in the 3D detector. We demonstrate both the efficacy and importance of these uncertainty estimates on identifying out-of-distribution scenes, poorly localized objects, and missing (false negative) detections.
  arXiv  Detail & Related papers  (2024-10-31T13:13:32Z)
• Automating Semantic Analysis of System Assurance Cases using Goal-directed ASP [1.2189422792863451]
  We present our approach to enhancing Assurance 2.0 with semantic rule-based analysis capabilities. We examine the unique semantic aspects of assurance cases, such as logical consistency, adequacy, indefeasibility, etc.
  arXiv  Detail & Related papers  (2024-08-21T15:22:43Z)
• LSM: A Comprehensive Metric for Assessing the Safety of Lane Detection Systems in Autonomous Driving [0.5326090003728084]
  We propose the Lane Safety Metric (LSM) to evaluate the safety of lane detection systems. Additional factors such as the semantics of the scene with road type and road width should be considered for the evaluation of lane detection. We evaluate our offline safety metric on various virtual scenarios using different lane detection approaches and compare it with state-of-the-art performance metrics.
  arXiv  Detail & Related papers  (2024-07-10T15:11:37Z)
• ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models [65.79770974145983]
  ASSERT, Automated Safety Scenario Red Teaming, consists of three methods -- semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. We find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings.
  arXiv  Detail & Related papers  (2023-10-14T17:10:28Z)
• DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection [55.70982767084996]
  A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. We present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions. DeepfakeBench contains 15 state-of-the-art detection methods, 9CL datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-07-04T01:34:41Z)
• Towards Building Self-Aware Object Detectors via Reliable Uncertainty Quantification and Calibration [17.461451218469062]
  In this work, we introduce the Self-Aware Object Detection (SAOD) task. The SAOD task respects and adheres to the challenges that object detectors face in safety-critical environments such as autonomous driving. We extensively use our framework, which introduces novel metrics and large scale test datasets, to test numerous object detectors.
  arXiv  Detail & Related papers  (2023-07-03T11:16:39Z)
• A Requirements-Driven Platform for Validating Field Operations of Small Uncrewed Aerial Vehicles [48.67061953896227]
  DroneReqValidator (DRV) allows sUAS developers to define the operating context, configure multi-sUAS mission requirements, specify safety properties, and deploy their own custom sUAS applications in a high-fidelity 3D environment. The DRV Monitoring system collects runtime data from sUAS and the environment, analyzes compliance with safety properties, and captures violations.
  arXiv  Detail & Related papers  (2023-07-01T02:03:49Z)
• Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key Features [5.634825161148484]
  This paper uses Instance Space Analysis (ISA) to identify the significant features of test scenarios that affect their ability to reveal the unsafe behaviour of AVs. ISA identifies the features that best differentiate safety-critical scenarios from normal driving and visualises the impact of these features on test scenario outcomes (safe/unsafe) in 2D. To test the predictive ability of the identified features, we train five Machine Learning classifiers to classify test scenarios as safe or unsafe.
  arXiv  Detail & Related papers  (2022-12-15T00:52:47Z)
• USC: Uncompromising Spatial Constraints for Safety-Oriented 3D Object Detectors in Autonomous Driving [7.355977594790584]
  We consider the safety-oriented performance of 3D object detectors in autonomous driving contexts. We present uncompromising spatial constraints (USC), which characterize a simple yet important localization requirement. We incorporate the quantitative measures into common loss functions to enable safety-oriented fine-tuning for existing models.
  arXiv  Detail & Related papers  (2022-09-21T14:03:08Z)
• CertainNet: Sampling-free Uncertainty Estimation for Object Detection [65.28989536741658]
  Estimating the uncertainty of a neural network plays a fundamental role in safety-critical settings. In this work, we propose a novel sampling-free uncertainty estimation method for object detection. We call it CertainNet, and it is the first to provide separate uncertainties for each output signal: objectness, class, location and size.
  arXiv  Detail & Related papers  (2021-10-04T17:59:31Z)
• Evaluating the Safety of Deep Reinforcement Learning Models using Semi-Formal Verification [81.32981236437395]
  We present a semi-formal verification approach for decision-making tasks based on interval analysis. Our method obtains comparable results over standard benchmarks with respect to formal verifiers. Our approach allows to efficiently evaluate safety properties for decision-making models in practical applications.
  arXiv  Detail & Related papers  (2020-10-19T11:18:06Z)
• Search-based Test-Case Generation by Monitoring Responsibility Safety Rules [2.1270496914042996]
  We propose a method for screening and classifying simulation-based driving test data to be used for training and testing controllers. Our framework is distributed with the publicly available S-TALIRO and Sim-ATAV tools.
  arXiv  Detail & Related papers  (2020-04-25T10:10:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.