System Safety Engineering for Social and Ethical ML Risks: A Case Study
- URL: http://arxiv.org/abs/2211.04602v1
- Date: Tue, 8 Nov 2022 22:58:58 GMT
- Title: System Safety Engineering for Social and Ethical ML Risks: A Case Study
- Authors: Edgar W. Jatho III and Logan O. Mailloux and Shalaleh Rismani and
Eugene D. Williams and Joshua A. Kroll
- Abstract summary: Governments, industry, and academia have undertaken efforts to identify and mitigate harms in ML-driven systems.
Existing approaches are largely disjointed, ad-hoc and of unknown effectiveness.
We focus in particular on how this analysis can extend to identifying social and ethical risks and developing concrete design-level controls to mitigate them.
- Score: 0.5249805590164902
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Governments, industry, and academia have undertaken efforts to identify and
mitigate harms in ML-driven systems, with a particular focus on social and
ethical risks of ML components in complex sociotechnical systems. However,
existing approaches are largely disjointed, ad-hoc and of unknown
effectiveness. Systems safety engineering is a well established discipline with
a track record of identifying and managing risks in many complex sociotechnical
domains. We adopt the natural hypothesis that tools from this domain could
serve to enhance risk analyses of ML in its context of use. To test this
hypothesis, we apply a "best of breed" systems safety analysis, Systems
Theoretic Process Analysis (STPA), to a specific high-consequence system with
an important ML-driven component, namely the Prescription Drug Monitoring
Programs (PDMPs) operated by many US States, several of which rely on an
ML-derived risk score. We focus in particular on how this analysis can extend
to identifying social and ethical risks and developing concrete design-level
controls to mitigate them.
Related papers
- Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal [0.0]
We propose a risk assessment process using tools like the risk rating methodology which is used for traditional systems.
We conduct scenario analysis to identify potential threat agents and map the dependent system components against vulnerability factors.
We also map threats against three key stakeholder groups.
arXiv Detail & Related papers (2024-03-20T05:17:22Z) - Prioritizing Safeguarding Over Autonomy: Risks of LLM Agents for Science [65.77763092833348]
Intelligent agents powered by large language models (LLMs) have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines.
While their capabilities are promising, these agents also introduce novel vulnerabilities that demand careful consideration for safety.
This paper conducts a thorough examination of vulnerabilities in LLM-based agents within scientific domains, shedding light on potential risks associated with their misuse and emphasizing the need for safety measures.
arXiv Detail & Related papers (2024-02-06T18:54:07Z) - Identifying the Risks of LM Agents with an LM-Emulated Sandbox [68.26587052548287]
Language Model (LM) agents and tools enable a rich set of capabilities but also amplify potential risks.
High cost of testing these agents will make it increasingly difficult to find high-stakes, long-tailed risks.
We introduce ToolEmu: a framework that uses an LM to emulate tool execution and enables the testing of LM agents against a diverse range of tools and scenarios.
arXiv Detail & Related papers (2023-09-25T17:08:02Z) - Vulnerability of Machine Learning Approaches Applied in IoT-based Smart Grid: A Review [51.31851488650698]
Machine learning (ML) sees an increasing prevalence of being used in the internet-of-things (IoT)-based smart grid.
adversarial distortion injected into the power signal will greatly affect the system's normal control and operation.
It is imperative to conduct vulnerability assessment for MLsgAPPs applied in the context of safety-critical power systems.
arXiv Detail & Related papers (2023-08-30T03:29:26Z) - Leveraging Traceability to Integrate Safety Analysis Artifacts into the
Software Development Process [51.42800587382228]
Safety assurance cases (SACs) can be challenging to maintain during system evolution.
We propose a solution that leverages software traceability to connect relevant system artifacts to safety analysis models.
We elicit design rationales for system changes to help safety stakeholders analyze the impact of system changes on safety.
arXiv Detail & Related papers (2023-07-14T16:03:27Z) - Concrete Safety for ML Problems: System Safety for ML Development and
Assessment [0.758305251912708]
Concerns of trustworthiness, unintended social harms, and unacceptable social and ethical violations undermine the promise of ML advancements.
Systems safety engineering is an established discipline with a proven track record of identifying and managing risks even in high-complexity sociotechnical systems.
arXiv Detail & Related papers (2023-02-06T18:02:07Z) - From plane crashes to algorithmic harm: applicability of safety
engineering frameworks for responsible ML [8.411124873373172]
Inappropriate design and deployment of machine learning (ML) systems leads to negative downstream social and ethical impact for users, society and the environment.
Despite the growing need to regulate ML systems, current processes for assessing and mitigating risks are disjointed and inconsistent.
arXiv Detail & Related papers (2022-10-06T00:09:06Z) - The Risks of Machine Learning Systems [11.105884571838818]
A system's overall risk is influenced by its direct and indirect effects.
Existing frameworks for ML risk/impact assessment often address an abstract notion of risk or do not concretize this dependence.
First-order risks stem from aspects of the ML system, while second-order risks stem from the consequences of first-order risks.
arXiv Detail & Related papers (2022-04-21T02:42:10Z) - A Framework for Evaluating the Cybersecurity Risk of Real World, Machine
Learning Production Systems [41.470634460215564]
We develop an extension to the MulVAL attack graph generation and analysis framework to incorporate cyberattacks on ML production systems.
Using the proposed extension, security practitioners can apply attack graph analysis methods in environments that include ML components.
arXiv Detail & Related papers (2021-07-05T05:58:11Z) - Practical Machine Learning Safety: A Survey and Primer [81.73857913779534]
Open-world deployment of Machine Learning algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities.
New models and training techniques to reduce generalization error, achieve domain adaptation, and detect outlier examples and adversarial attacks.
Our organization maps state-of-the-art ML techniques to safety strategies in order to enhance the dependability of the ML algorithm from different aspects.
arXiv Detail & Related papers (2021-06-09T05:56:42Z) - Inspect, Understand, Overcome: A Survey of Practical Methods for AI
Safety [54.478842696269304]
The use of deep neural networks (DNNs) in safety-critical applications is challenging due to numerous model-inherent shortcomings.
In recent years, a zoo of state-of-the-art techniques aiming to address these safety concerns has emerged.
Our paper addresses both machine learning experts and safety engineers.
arXiv Detail & Related papers (2021-04-29T09:54:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.