Incorporating Failure of Machine Learning in Dynamic Probabilistic Safety Assurance

• URL: http://arxiv.org/abs/2506.06868v1
• Date: Sat, 07 Jun 2025 17:16:05 GMT
• Title: Incorporating Failure of Machine Learning in Dynamic Probabilistic Safety Assurance
• Authors: Razieh Arshadizadeh, Mahmoud Asgari, Zeinab Khosravi, Yiannis Papadopoulos, Koorosh Aslansefat,
• Abstract summary: We introduce a safety assurance framework that integrates SafeML with Bayesian Networks (BNs) to model ML failures as part of a broader causal safety analysis.<n>We demonstrate the approach on a simulated automotive platooning system with traffic sign recognition.
• Score: 0.1398098625978622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine Learning (ML) models are increasingly integrated into safety-critical systems, such as autonomous vehicle platooning, to enable real-time decision-making. However, their inherent imperfection introduces a new class of failure: reasoning failures often triggered by distributional shifts between operational and training data. Traditional safety assessment methods, which rely on design artefacts or code, are ill-suited for ML components that learn behaviour from data. SafeML was recently proposed to dynamically detect such shifts and assign confidence levels to the reasoning of ML-based components. Building on this, we introduce a probabilistic safety assurance framework that integrates SafeML with Bayesian Networks (BNs) to model ML failures as part of a broader causal safety analysis. This allows for dynamic safety evaluation and system adaptation under uncertainty. We demonstrate the approach on an simulated automotive platooning system with traffic sign recognition. The findings highlight the potential broader benefits of explicitly modelling ML failures in safety assessment.

Related papers

• Robust and Safe Traffic Sign Recognition using N-version with Weighted Voting [5.087405608698016]
  We propose an N-version machine learning (NVML) framework that integrates a safety-aware weighted soft voting mechanism.<n>Our approach utilizes Failure Mode and Effects Analysis (FMEA) to assess potential safety risks and assign dynamic, safety-aware weights to the ensemble outputs.<n> Experimental results demonstrate that our NVML approach significantly enhances the robustness and safety of traffic sign recognition systems under adversarial conditions.
  arXiv  Detail & Related papers  (2025-07-09T14:46:31Z)
• SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models [50.34706204154244]
  Acquiring reasoning capabilities catastrophically degrades inherited safety alignment.<n>Certain scenarios suffer 25 times higher attack rates.<n>Despite tight reasoning-answer safety coupling, MLRMs demonstrate nascent self-correction.
  arXiv  Detail & Related papers  (2025-04-09T06:53:23Z)
• Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Models [92.38300626647342]
  Fine-tuning Large Language Models (LLMs) on some task-specific datasets has been a primary use of LLMs.<n>This paper presents a theoretical framework for understanding the interplay between safety and capability in two primary safety-aware LLM fine-tuning strategies.
  arXiv  Detail & Related papers  (2025-03-24T20:41:57Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signals [50.463399903987245]
  Large language models (LLMs) exhibit exceptional capabilities across various tasks but also pose risks by generating harmful content.<n>We show that LLMs can similarly perform internal assessments about safety in their internal states.<n>We propose SafeSwitch, a framework that regulates unsafe outputs by utilizing the prober-based internal state monitor.
  arXiv  Detail & Related papers  (2025-02-03T04:23:33Z)
• Incorporating System-level Safety Requirements in Perception Models via Reinforcement Learning [7.833541053347799]
  We propose a training paradigm that augments the perception component with an understanding of system-level safety objectives.<n>We show that models trained with this approach outperform baseline perception models in terms of system-level safety.
  arXiv  Detail & Related papers  (2024-12-04T01:40:54Z)
• SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models [75.67623347512368]
  We propose toolns, a comprehensive framework designed for conducting safety evaluations of MLLMs. Our framework consists of a comprehensive harmful query dataset and an automated evaluation protocol. Based on our framework, we conducted large-scale experiments on 15 widely-used open-source MLLMs and 6 commercial MLLMs.
  arXiv  Detail & Related papers  (2024-10-24T17:14:40Z)
• Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training [67.30423823744506]
  We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position.<n>DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence.
  arXiv  Detail & Related papers  (2024-07-12T09:36:33Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)
• Dynamic Model Agnostic Reliability Evaluation of Machine-Learning Methods Integrated in Instrumentation & Control Systems [1.8978726202765634]
  Trustworthiness of datadriven neural network-based machine learning algorithms is not adequately assessed. In recent reports by the National Institute for Standards and Technology, trustworthiness in ML is a critical barrier to adoption. We demonstrate a real-time model-agnostic method to evaluate the relative reliability of ML predictions by incorporating out-of-distribution detection on the training dataset.
  arXiv  Detail & Related papers  (2023-08-08T18:25:42Z)
• Uncertainty-aware predictive modeling for fair data-driven decisions [5.371337604556311]
  We show how fairML systems can be safeML systems. For fair decisions, we argue that a safe fail option should be used for individuals with uncertain categorization.
  arXiv  Detail & Related papers  (2022-11-04T20:04:39Z)
• Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
  We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers. We then present the pointwise feasibility conditions of the resulting safety controller. We use these conditions to devise an event-triggered online data collection strategy.
  arXiv  Detail & Related papers  (2022-08-23T05:02:09Z)
• Reliability Assessment and Safety Arguments for Machine Learning Components in Assuring Learning-Enabled Autonomous Systems [19.65793237440738]
  We present an overall assurance framework for Learning-Enabled Systems (LES) We then introduce a novel model-agnostic Reliability Assessment Model (RAM) for ML classifiers. We discuss the model assumptions and the inherent challenges of assessing ML reliability uncovered by our RAM.
  arXiv  Detail & Related papers  (2021-11-30T14:39:22Z)
• Towards Probability-based Safety Verification of Systems with Components from Machine Learning [8.75682288556859]
  Safety verification of machine learning systems is currently thought to be infeasible or, at least, very hard. We think that it requires taking into account specific properties of ML technology such as: (i) Most ML approaches are inductive, which is both their power and their source of error. We propose verification based on probabilities of errors both estimated by controlled experiments and output by the inductively learned itself.
  arXiv  Detail & Related papers  (2020-03-02T19:31:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.