SafeML: Safety Monitoring of Machine Learning Classifiers through Statistical Difference Measure

• URL: http://arxiv.org/abs/2005.13166v1
• Date: Wed, 27 May 2020 05:27:38 GMT
• Title: SafeML: Safety Monitoring of Machine Learning Classifiers through Statistical Difference Measure
• Authors: Koorosh Aslansefat, Ioannis Sorokos, Declan Whiting, Ramin Tavakoli Kolagari, Yiannis Papadopoulos
• Abstract summary: This paper aims to address both safety and security within a single concept of protection applicable during the operation of ML systems. We use distance measures of the Empirical Cumulative Distribution Function (ECDF) to monitor the behaviour and the operational context of the data-driven system. Our preliminary findings indicate that the approach can provide a basis for detecting whether the application context of an ML component is valid in the safety-security.
• Score: 1.2599533416395765
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Ensuring safety and explainability of machine learning (ML) is a topic of increasing relevance as data-driven applications venture into safety-critical application domains, traditionally committed to high safety standards that are not satisfied with an exclusive testing approach of otherwise inaccessible black-box systems. Especially the interaction between safety and security is a central challenge, as security violations can lead to compromised safety. The contribution of this paper to addressing both safety and security within a single concept of protection applicable during the operation of ML systems is active monitoring of the behaviour and the operational context of the data-driven system based on distance measures of the Empirical Cumulative Distribution Function (ECDF). We investigate abstract datasets (XOR, Spiral, Circle) and current security-specific datasets for intrusion detection (CICIDS2017) of simulated network traffic, using distributional shift detection measures including the Kolmogorov-Smirnov, Kuiper, Anderson-Darling, Wasserstein and mixed Wasserstein-Anderson-Darling measures. Our preliminary findings indicate that the approach can provide a basis for detecting whether the application context of an ML component is valid in the safety-security. Our preliminary code and results are available at https://github.com/ISorokos/SafeML.

Related papers

• Nothing in Excess: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering [56.92068213969036]
  Safety alignment is indispensable for Large language models (LLMs) to defend threats from malicious instructions. Recent researches reveal safety-aligned LLMs prone to reject benign queries due to the exaggerated safety issue. We propose a Safety-Conscious Activation Steering (SCANS) method to mitigate the exaggerated safety concerns.
  arXiv  Detail & Related papers  (2024-08-21T10:01:34Z)
• Cross-Modality Safety Alignment [73.8765529028288]
  We introduce a novel safety alignment challenge called Safe Inputs but Unsafe Output (SIUO) to evaluate cross-modality safety alignment. To empirically investigate this problem, we developed the SIUO, a cross-modality benchmark encompassing 9 critical safety domains, such as self-harm, illegal activities, and privacy violations. Our findings reveal substantial safety vulnerabilities in both closed- and open-source LVLMs, underscoring the inadequacy of current models to reliably interpret and respond to complex, real-world scenarios.
  arXiv  Detail & Related papers  (2024-06-21T16:14:15Z)
• System Safety Monitoring of Learned Components Using Temporal Metric Forecasting [8.76735390039138]
  In learning-enabled autonomous systems, safety monitoring of learned components is crucial to ensure their outputs do not lead to system safety violations. We propose a safety monitoring method based on probabilistic time series forecasting. We empirically evaluate safety metric and violation prediction accuracy, and inference latency and resource usage of four state-of-the-art models.
  arXiv  Detail & Related papers  (2024-05-21T23:48:26Z)
• Vulnerability of Machine Learning Approaches Applied in IoT-based Smart Grid: A Review [51.31851488650698]
  Machine learning (ML) sees an increasing prevalence of being used in the internet-of-things (IoT)-based smart grid. adversarial distortion injected into the power signal will greatly affect the system's normal control and operation. It is imperative to conduct vulnerability assessment for MLsgAPPs applied in the context of safety-critical power systems.
  arXiv  Detail & Related papers  (2023-08-30T03:29:26Z)
• Leveraging Traceability to Integrate Safety Analysis Artifacts into the Software Development Process [51.42800587382228]
  Safety assurance cases (SACs) can be challenging to maintain during system evolution. We propose a solution that leverages software traceability to connect relevant system artifacts to safety analysis models. We elicit design rationales for system changes to help safety stakeholders analyze the impact of system changes on safety.
  arXiv  Detail & Related papers  (2023-07-14T16:03:27Z)
• Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
  We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers. We then present the pointwise feasibility conditions of the resulting safety controller. We use these conditions to devise an event-triggered online data collection strategy.
  arXiv  Detail & Related papers  (2022-08-23T05:02:09Z)
• Fail-Safe Adversarial Generative Imitation Learning [9.594432031144716]
  We propose a safety layer that enables a closed-form probability density/gradient of the safe generative continuous policy, end-to-end generative adversarial training, and worst-case safety guarantees. The safety layer maps all actions into a set of safe actions, and uses the change-of-variables formula plus additivity of measures for the density. In an experiment on real-world driver interaction data, we empirically demonstrate tractability, safety and imitation performance of our approach.
  arXiv  Detail & Related papers  (2022-03-03T13:03:06Z)
• MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance [73.3242641337305]
  Recent work learns risk measures which measure the probability of violating constraints, which can then be used to enable safety. We cast safe exploration as an offline meta-RL problem, where the objective is to leverage examples of safe and unsafe behavior across a range of environments. We then propose MEta-learning for Safe Adaptation (MESA), an approach for meta-learning Simulation a risk measure for safe RL.
  arXiv  Detail & Related papers  (2021-12-07T08:57:35Z)
• Reliability Assessment and Safety Arguments for Machine Learning Components in Assuring Learning-Enabled Autonomous Systems [19.65793237440738]
  We present an overall assurance framework for Learning-Enabled Systems (LES) We then introduce a novel model-agnostic Reliability Assessment Model (RAM) for ML classifiers. We discuss the model assumptions and the inherent challenges of assessing ML reliability uncovered by our RAM.
  arXiv  Detail & Related papers  (2021-11-30T14:39:22Z)
• A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems [30.638615396429536]
  Motivated by the prevalence of safety-critical artificial intelligence, this work provides a survey of state-of-the-art safety validation techniques for CPS. We present and discuss algorithms in the domains of optimization, path planning, reinforcement learning, and importance sampling. A brief overview of safety-critical applications is given, including autonomous vehicles and aircraft collision avoidance systems.
  arXiv  Detail & Related papers  (2020-05-06T17:31:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.