Explaining Unreliable Perception in Automated Driving: A Fuzzy-based Monitoring Approach

• URL: http://arxiv.org/abs/2505.14407v1
• Date: Tue, 20 May 2025 14:22:39 GMT
• Title: Explaining Unreliable Perception in Automated Driving: A Fuzzy-based Monitoring Approach
• Authors: Aniket Salvi, Gereon Weiss, Mario Trapp,
• Abstract summary: This paper introduces a novel fuzzy-based monitor tailored for Machine Learning (ML) perception components.<n>It provides human-interpretable explanations about how different operating conditions affect the reliability of perception components and also functions as a runtime safety monitor.
• Score: 1.33134751838052
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous systems that rely on Machine Learning (ML) utilize online fault tolerance mechanisms, such as runtime monitors, to detect ML prediction errors and maintain safety during operation. However, the lack of human-interpretable explanations for these errors can hinder the creation of strong assurances about the system's safety and reliability. This paper introduces a novel fuzzy-based monitor tailored for ML perception components. It provides human-interpretable explanations about how different operating conditions affect the reliability of perception components and also functions as a runtime safety monitor. We evaluated our proposed monitor using naturalistic driving datasets as part of an automated driving case study. The interpretability of the monitor was evaluated and we identified a set of operating conditions in which the perception component performs reliably. Additionally, we created an assurance case that links unit-level evidence of \textit{correct} ML operation to system-level \textit{safety}. The benchmarking demonstrated that our monitor achieved a better increase in safety (i.e., absence of hazardous situations) while maintaining availability (i.e., ability to perform the mission) compared to state-of-the-art runtime ML monitors in the evaluated dataset.

Related papers

• Verification of Visual Controllers via Compositional Geometric Transformations [49.81690518952909]
  We introduce a novel verification framework for perception-based controllers that can generate outer-approximations of reachable sets.<n>We provide theoretical guarantees on the soundness of our method and demonstrate its effectiveness across benchmark control environments.
  arXiv  Detail & Related papers  (2025-07-06T20:22:58Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  Multimodal Large Language Models (MLLMs) can process both visual and textual data.<n>We propose SafeAuto, a novel framework that enhances MLLM-based autonomous driving systems by incorporating both unstructured and structured knowledge.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Safety Monitoring of Machine Learning Perception Functions: a Survey [7.193217430660011]
  New dependability challenges arise when Machine Learning predictions are used in safety-critical applications.<n>The use of fault tolerance mechanisms, such as safety monitors, is essential to ensure the safe behavior of the system.<n>This paper presents an extensive literature review on safety monitoring of perception functions using ML in a safety-critical context.
  arXiv  Detail & Related papers  (2024-12-09T10:58:50Z)
• Learning Run-time Safety Monitors for Machine Learning Components [8.022333445774382]
  This paper introduces a process for creating safety monitors for machine learning components through the use of degraded datasets and machine learning. The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output.
  arXiv  Detail & Related papers  (2024-06-23T21:25:06Z)
• System Safety Monitoring of Learned Components Using Temporal Metric Forecasting [8.76735390039138]
  In learning-enabled autonomous systems, safety monitoring of learned components is crucial to ensure their outputs do not lead to system safety violations.<n>We propose a safety monitoring method based on probabilistic time series forecasting.<n>We empirically evaluate safety metric and violation prediction accuracy, and inference latency and resource usage of four state-of-the-art models.
  arXiv  Detail & Related papers  (2024-05-21T23:48:26Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)
• Unifying Evaluation of Machine Learning Safety Monitors [0.0]
  runtime monitors have been developed to detect prediction errors and keep the system in a safe state during operations. This paper introduces three unified safety-oriented metrics, representing the safety benefits of the monitor (Safety Gain) and the remaining safety gaps after using it (Residual Hazard) Three use-cases (classification, drone landing, and autonomous driving) are used to demonstrate how metrics from the literature can be expressed in terms of the proposed metrics.
  arXiv  Detail & Related papers  (2022-08-31T07:17:42Z)
• Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
  We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers. We then present the pointwise feasibility conditions of the resulting safety controller. We use these conditions to devise an event-triggered online data collection strategy.
  arXiv  Detail & Related papers  (2022-08-23T05:02:09Z)
• Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations [50.37808220291108]
  This paper addresses learning safe output feedback control laws from partial observations of expert demonstrations. We first propose robust output control barrier functions (ROCBFs) as a means to guarantee safety. We then formulate an optimization problem to learn ROCBFs from expert demonstrations that exhibit safe system behavior.
  arXiv  Detail & Related papers  (2021-11-18T23:21:00Z)
• Benchmarking Safety Monitors for Image Classifiers with Machine Learning [0.0]
  High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation. The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state. This paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers.
  arXiv  Detail & Related papers  (2021-10-04T07:52:23Z)
• Sample-Efficient Safety Assurances using Conformal Prediction [57.92013073974406]
  Early warning systems can provide alerts when an unsafe situation is imminent. To reliably improve safety, these warning systems should have a provable false negative rate. We present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics.
  arXiv  Detail & Related papers  (2021-09-28T23:00:30Z)
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [77.39239190539871]
  We show how uncertainty estimation can be leveraged to enable safety critical image segmentation in autonomous driving. We introduce a new uncertainty measure based on disagreeing predictions as measured by a dissimilarity function. We show experimentally that our proposed approach is much less computationally intensive at inference time than competing methods.
  arXiv  Detail & Related papers  (2021-05-28T09:23:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.