Toward Comprehensive Risk Assessments and Assurance of AI-Based Systems

• URL: http://arxiv.org/abs/2606.29390v1
• Date: Sun, 28 Jun 2026 13:26:03 GMT
• Title: Toward Comprehensive Risk Assessments and Assurance of AI-Based Systems
• Abstract summary: Safety, socio-economic, and ethical harms arising from the deployment of AI-based systems have led to a breadth of work seeking to map, measure, and mitigate against newly found risks.<n>Previous attempts to adapt risk assessment techniques and terminology from the safety and security fields have been insufficient.<n>We propose a novel end-to-end AI risk framework that integrates the concept of an Operational Design Domains (ODD) for more general AI-based systems.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Novel safety, socio-economic, and ethical harms arising from the deployment of AI-based systems have led to a breadth of work seeking to map, measure, and mitigate against newly found risks. These works have heavily leveraged techniques and terminology from the fields of System Safety Engineering and Cybersecurity, yet they have fallen short in accounting for the limitations and nuances that reduce the efficacy and correct application of adopted methodologies. Furthermore, misuse of terminology entailing compliance with established safety and security properties can mislead stakeholders with regard to the claims an AI system satisfies and provide a false sense of safety. In this paper, we seek to align overlapping, AI-adjacent communities on a consistent and comprehensive assurance terminology crucial for the safe deployment of AI-based systems. We outline why previous attempts to adapt risk assessment techniques and terminology from the safety and security fields have been insufficient. We then propose a novel end-to-end AI risk framework that integrates the concept of an Operational Design Domains (ODD), initially introduced for ADS (Automated Driving Systems) [1], for more general AI-based systems. The purpose of an ODD is to provide a description of the specific operating conditions for which an AI-system is designed to properly behave, thus outlining the safety envelope for which system hazards and harms can be determined against. We believe that by defining a more concrete operational envelope, developers and auditors can better assess potential risks and required safety mitigations for AI-based systems.

Related papers

• Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies [57.521647436515785]
  We define frontier AI auditing as rigorous third-party verification of frontier AI developers' safety and security claims.<n>We introduce AI Assurance Levels (AAL-1 to AAL-4), ranging from time-bounded system audits to continuous, deception-resilient verification.
  arXiv  Detail & Related papers  (2026-01-16T18:44:09Z)
• Towards provable probabilistic safety for scalable embodied AI systems [79.31011047593492]
  Embodied AI systems are increasingly prevalent across various applications.<n> Ensuring their safety in complex operating environments remains a major challenge.<n>This Perspective offers a pathway toward safer, large-scale adoption of embodied AI systems in safety-critical applications.
  arXiv  Detail & Related papers  (2025-06-05T15:46:25Z)
• Systematic Hazard Analysis for Frontier AI using STPA [0.0]
  frontier AI companies currently do not describe in detail any structured approach to identifying and analysing hazards.<n>A (Systems-Theoretic Process Analysis) is a systematic methodology for identifying how complex systems can become unsafe, leading to hazards.<n>We evaluateA's ability to broaden the scope, improve traceability and strengthen the robustness of safety assurance for frontier AI systems.
  arXiv  Detail & Related papers  (2025-06-02T15:28:34Z)
• Safety Co-Option and Compromised National Security: The Self-Fulfilling Prophecy of Weakened AI Risk Thresholds [0.0]
  We show how "safety revisionism" has allowed AI technologists to engage in "safety revisionism"<n>We explore how the current trajectory for AI risk determination and evaluation for foundation model use within national security is poised for a race to the bottom.
  arXiv  Detail & Related papers  (2025-04-21T13:20:56Z)
• An Approach to Technical AGI Safety and Security [72.83728459135101]
  We develop an approach to address the risk of harms consequential enough to significantly harm humanity.<n>We focus on technical approaches to misuse and misalignment.<n>We briefly outline how these ingredients could be combined to produce safety cases for AGI systems.
  arXiv  Detail & Related papers  (2025-04-02T15:59:31Z)
• Position: Mind the Gap-the Growing Disconnect Between Established Vulnerability Disclosure and AI Security [56.219994752894294]
  We argue that adapting existing processes for AI security reporting is doomed to fail due to fundamental shortcomings for the distinctive characteristics of AI systems.<n>Based on our proposal to address these shortcomings, we discuss an approach to AI security reporting and how the new AI paradigm, AI agents, will further reinforce the need for specialized AI security incident reporting advancements.
  arXiv  Detail & Related papers  (2024-12-19T13:50:26Z)
• Landscape of AI safety concerns -- A methodology to support safety assurance for AI-based autonomous systems [0.0]
  AI has emerged as a key technology, driving advancements across a range of applications.<n>The challenge of assuring safety in systems that incorporate AI components is substantial.<n>We propose a novel methodology designed to support the creation of safety assurance cases for AI-based systems.
  arXiv  Detail & Related papers  (2024-12-18T16:38:16Z)
• Safeguarding AI Agents: Developing and Analyzing Safety Architectures [0.0]
  This paper addresses the need for safety measures in AI systems that collaborate with human teams.<n>We propose and evaluate three frameworks to enhance safety protocols in AI agent systems.<n>We conclude that these frameworks can significantly strengthen the safety and security of AI agent systems.
  arXiv  Detail & Related papers  (2024-09-03T10:14:51Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction.<n>Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results.<n>However, the deployment of these agents in physical environments presents significant safety challenges.<n>This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)
• Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems [88.80306881112313]
  We will introduce and define a family of approaches to AI safety, which we will refer to as guaranteed safe (GS) AI. The core feature of these approaches is that they aim to produce AI systems which are equipped with high-assurance quantitative safety guarantees. We outline a number of approaches for creating each of these three core components, describe the main technical challenges, and suggest a number of potential solutions to them.
  arXiv  Detail & Related papers  (2024-05-10T17:38:32Z)
• System Safety and Artificial Intelligence [0.0]
  New applications of AI across societal domains come with new hazards. The field of system safety has dealt with accidents and harm in safety-critical systems. This chapter honors system safety pioneer Nancy Leveson.
  arXiv  Detail & Related papers  (2022-02-18T16:37:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.