Learning Run-time Safety Monitors for Machine Learning Components

• URL: http://arxiv.org/abs/2406.16220v1
• Date: Sun, 23 Jun 2024 21:25:06 GMT
• Title: Learning Run-time Safety Monitors for Machine Learning Components
• Authors: Ozan Vardal, Richard Hawkins, Colin Paterson, Chiara Picardi, Daniel Omeiza, Lars Kunze, Ibrahim Habli,
• Abstract summary: This paper introduces a process for creating safety monitors for machine learning components through the use of degraded datasets and machine learning. The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output.
• Score: 8.022333445774382
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: For machine learning components used as part of autonomous systems (AS) in carrying out critical tasks it is crucial that assurance of the models can be maintained in the face of post-deployment changes (such as changes in the operating environment of the system). A critical part of this is to be able to monitor when the performance of the model at runtime (as a result of changes) poses a safety risk to the system. This is a particularly difficult challenge when ground truth is unavailable at runtime. In this paper we introduce a process for creating safety monitors for ML components through the use of degraded datasets and machine learning. The safety monitor that is created is deployed to the AS in parallel to the ML component to provide a prediction of the safety risk associated with the model output. We demonstrate the viability of our approach through some initial experiments using publicly available speed sign datasets.

Related papers

• System Safety Monitoring of Learned Components Using Temporal Metric Forecasting [8.76735390039138]
  In learning-enabled autonomous systems, safety monitoring of learned components is crucial to ensure their outputs do not lead to system safety violations. We propose a safety monitoring method based on probabilistic time series forecasting. We empirically evaluate safety metric and violation prediction accuracy, and inference latency and resource usage of four state-of-the-art models.
  arXiv  Detail & Related papers  (2024-05-21T23:48:26Z)
• Highlighting the Safety Concerns of Deploying LLMs/VLMs in Robotics [54.57914943017522]
  We highlight the critical issues of robustness and safety associated with integrating large language models (LLMs) and vision-language models (VLMs) into robotics applications.
  arXiv  Detail & Related papers  (2024-02-15T22:01:45Z)
• Assurance for Deployed Continual Learning Systems [0.0]
  The authors created a new framework for safely performing continual learning with a deep learning computer vision algorithm. The safety framework includes several features, such as an ensemble of convolutional neural networks to perform image classification. The results also show the framework can detect when the system is no longer performing safely.
  arXiv  Detail & Related papers  (2023-11-16T22:22:13Z)
• Privacy Side Channels in Machine Learning Systems [87.53240071195168]
  We introduce privacy side channels: attacks that exploit system-level components to extract private information. For example, we show that deduplicating training data before applying differentially-private training creates a side-channel that completely invalidates any provable privacy guarantees. We further show that systems which block language models from regenerating training data can be exploited to exfiltrate private keys contained in the training set.
  arXiv  Detail & Related papers  (2023-09-11T16:49:05Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)
• Unifying Evaluation of Machine Learning Safety Monitors [0.0]
  runtime monitors have been developed to detect prediction errors and keep the system in a safe state during operations. This paper introduces three unified safety-oriented metrics, representing the safety benefits of the monitor (Safety Gain) and the remaining safety gaps after using it (Residual Hazard) Three use-cases (classification, drone landing, and autonomous driving) are used to demonstrate how metrics from the literature can be expressed in terms of the proposed metrics.
  arXiv  Detail & Related papers  (2022-08-31T07:17:42Z)
• Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
  We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers. We then present the pointwise feasibility conditions of the resulting safety controller. We use these conditions to devise an event-triggered online data collection strategy.
  arXiv  Detail & Related papers  (2022-08-23T05:02:09Z)
• An Empirical Analysis of the Use of Real-Time Reachability for the Safety Assurance of Autonomous Vehicles [7.1169864450668845]
  We propose using a real-time reachability algorithm for the implementation of the simplex architecture to assure the safety of a 1/10 scale open source autonomous vehicle platform. In our approach, the need to analyze an underlying controller is abstracted away, instead focusing on the effects of the controller's decisions on the system's future states.
  arXiv  Detail & Related papers  (2022-05-03T11:12:29Z)
• Benchmarking Safety Monitors for Image Classifiers with Machine Learning [0.0]
  High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation. The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state. This paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers.
  arXiv  Detail & Related papers  (2021-10-04T07:52:23Z)
• Multi Agent System for Machine Learning Under Uncertainty in Cyber Physical Manufacturing System [78.60415450507706]
  Recent advancements in predictive machine learning has led to its application in various use cases in manufacturing. Most research focused on maximising predictive accuracy without addressing the uncertainty associated with it. In this paper, we determine the sources of uncertainty in machine learning and establish the success criteria of a machine learning system to function well under uncertainty.
  arXiv  Detail & Related papers  (2021-07-28T10:28:05Z)
• Dos and Don'ts of Machine Learning in Computer Security [74.1816306998445]
  Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance. We identify common pitfalls in the design, implementation, and evaluation of learning-based security systems. We propose actionable recommendations to support researchers in avoiding or mitigating the pitfalls where possible.
  arXiv  Detail & Related papers  (2020-10-19T13:09:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.