System Safety Engineering for Social and Ethical ML Risks: A Case Study

• URL: http://arxiv.org/abs/2211.04602v1
• Date: Tue, 8 Nov 2022 22:58:58 GMT
• Title: System Safety Engineering for Social and Ethical ML Risks: A Case Study
• Authors: Edgar W. Jatho III and Logan O. Mailloux and Shalaleh Rismani and Eugene D. Williams and Joshua A. Kroll
• Abstract summary: Governments, industry, and academia have undertaken efforts to identify and mitigate harms in ML-driven systems. Existing approaches are largely disjointed, ad-hoc and of unknown effectiveness. We focus in particular on how this analysis can extend to identifying social and ethical risks and developing concrete design-level controls to mitigate them.
• Score: 0.5249805590164902
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Governments, industry, and academia have undertaken efforts to identify and mitigate harms in ML-driven systems, with a particular focus on social and ethical risks of ML components in complex sociotechnical systems. However, existing approaches are largely disjointed, ad-hoc and of unknown effectiveness. Systems safety engineering is a well established discipline with a track record of identifying and managing risks in many complex sociotechnical domains. We adopt the natural hypothesis that tools from this domain could serve to enhance risk analyses of ML in its context of use. To test this hypothesis, we apply a "best of breed" systems safety analysis, Systems Theoretic Process Analysis (STPA), to a specific high-consequence system with an important ML-driven component, namely the Prescription Drug Monitoring Programs (PDMPs) operated by many US States, several of which rely on an ML-derived risk score. We focus in particular on how this analysis can extend to identifying social and ethical risks and developing concrete design-level controls to mitigate them.

Related papers

• Multi-Agent Risks from Advanced AI [90.74347101431474]
  Multi-agent systems of advanced AI pose novel and under-explored risks. We identify three key failure modes based on agents' incentives, as well as seven key risk factors. We highlight several important instances of each risk, as well as promising directions to help mitigate them.
  arXiv  Detail & Related papers  (2025-02-19T23:03:21Z)
• From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems [17.496832430021968]
  Machine learning (ML) components are increasingly integrated into software products. Despite these risks, practitioners seldom adopt proactive approaches to anticipate and mitigate hazards. This position paper advocates for integrating hazard analysis into the development of any ML-powered software product.
  arXiv  Detail & Related papers  (2025-02-11T21:37:19Z)
• Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents [67.07177243654485]
  This survey collects and analyzes the different threats faced by large language models-based agents. We identify six key features of LLM-based agents, based on which we summarize the current research progress. We select four representative agents as case studies to analyze the risks they may face in practical use.
  arXiv  Detail & Related papers  (2024-11-14T15:40:04Z)
• SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models [75.67623347512368]
  We propose toolns, a comprehensive framework designed for conducting safety evaluations of MLLMs. Our framework consists of a comprehensive harmful query dataset and an automated evaluation protocol. Based on our framework, we conducted large-scale experiments on 15 widely-used open-source MLLMs and 6 commercial MLLMs.
  arXiv  Detail & Related papers  (2024-10-24T17:14:40Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction. Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results. However, the deployment of these agents in physical environments presents significant safety challenges. This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)
• Prioritizing Safeguarding Over Autonomy: Risks of LLM Agents for Science [65.77763092833348]
  Intelligent agents powered by large language models (LLMs) have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines. While their capabilities are promising, these agents also introduce novel vulnerabilities that demand careful consideration for safety. This paper conducts a thorough examination of vulnerabilities in LLM-based agents within scientific domains, shedding light on potential risks associated with their misuse and emphasizing the need for safety measures.
  arXiv  Detail & Related papers  (2024-02-06T18:54:07Z)
• Leveraging Traceability to Integrate Safety Analysis Artifacts into the Software Development Process [51.42800587382228]
  Safety assurance cases (SACs) can be challenging to maintain during system evolution. We propose a solution that leverages software traceability to connect relevant system artifacts to safety analysis models. We elicit design rationales for system changes to help safety stakeholders analyze the impact of system changes on safety.
  arXiv  Detail & Related papers  (2023-07-14T16:03:27Z)
• Concrete Safety for ML Problems: System Safety for ML Development and Assessment [0.758305251912708]
  Concerns of trustworthiness, unintended social harms, and unacceptable social and ethical violations undermine the promise of ML advancements. Systems safety engineering is an established discipline with a proven track record of identifying and managing risks even in high-complexity sociotechnical systems.
  arXiv  Detail & Related papers  (2023-02-06T18:02:07Z)
• From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML [8.411124873373172]
  Inappropriate design and deployment of machine learning (ML) systems leads to negative downstream social and ethical impact for users, society and the environment. Despite the growing need to regulate ML systems, current processes for assessing and mitigating risks are disjointed and inconsistent.
  arXiv  Detail & Related papers  (2022-10-06T00:09:06Z)
• Quantitative AI Risk Assessments: Opportunities and Challenges [7.35411010153049]
  Best way to reduce risks is to implement comprehensive AI lifecycle governance. Risks can be quantified using metrics from the technical community. This paper explores these issues, focusing on the opportunities, challenges, and potential impacts of such an approach.
  arXiv  Detail & Related papers  (2022-09-13T21:47:25Z)
• The Risks of Machine Learning Systems [11.105884571838818]
  A system's overall risk is influenced by its direct and indirect effects. Existing frameworks for ML risk/impact assessment often address an abstract notion of risk or do not concretize this dependence. First-order risks stem from aspects of the ML system, while second-order risks stem from the consequences of first-order risks.
  arXiv  Detail & Related papers  (2022-04-21T02:42:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.