Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance

• URL: http://arxiv.org/abs/2009.00802v1
• Date: Wed, 2 Sep 2020 03:33:40 GMT
• Title: Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance
• Authors: Andrew J. Lohn
• Abstract summary: Test, Evaluation, Verification, and Validation for Artificial Intelligence (AI) is a challenge that threatens to limit the economic and societal rewards that AI researchers have devoted themselves to producing. This paper argues that neither of those criteria are certain of Deep Neural Networks.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test, Evaluation, Verification, and Validation (TEVV) for Artificial Intelligence (AI) is a challenge that threatens to limit the economic and societal rewards that AI researchers have devoted themselves to producing. A central task of TEVV for AI is estimating brittleness, where brittleness implies that the system functions well within some bounds and poorly outside of those bounds. This paper argues that neither of those criteria are certain of Deep Neural Networks. First, highly touted AI successes (eg. image classification and speech recognition) are orders of magnitude more failure-prone than are typically certified in critical systems even within design bounds (perfectly in-distribution sampling). Second, performance falls off only gradually as inputs become further Out-Of-Distribution (OOD). Enhanced emphasis is needed on designing systems that are resilient despite failure-prone AI components as well as on evaluating and improving OOD performance in order to get AI to where it can clear the challenging hurdles of TEVV and certification.

Related papers

• Imagining and building wise machines: The centrality of AI metacognition [78.76893632793497]
  We argue that shortcomings stem from one overarching failure: AI systems lack wisdom. While AI research has focused on task-level strategies, metacognition is underdeveloped in AI systems. We propose that integrating metacognitive capabilities into AI systems is crucial for enhancing their robustness, explainability, cooperation, and safety.
  arXiv  Detail & Related papers  (2024-11-04T18:10:10Z)
• Engineering Trustworthy AI: A Developer Guide for Empirical Risk Minimization [53.80919781981027]
  Key requirements for trustworthy AI can be translated into design choices for the components of empirical risk minimization. We hope to provide actionable guidance for building AI systems that meet emerging standards for trustworthiness of AI.
  arXiv  Detail & Related papers  (2024-10-25T07:53:32Z)
• Using AI Alignment Theory to understand the potential pitfalls of regulatory frameworks [55.2480439325792]
  This paper critically examines the European Union's Artificial Intelligence Act (EU AI Act) Uses insights from Alignment Theory (AT) research, which focuses on the potential pitfalls of technical alignment in Artificial Intelligence. As we apply these concepts to the EU AI Act, we uncover potential vulnerabilities and areas for improvement in the regulation.
  arXiv  Detail & Related papers  (2024-10-10T17:38:38Z)
• AI Sandbagging: Language Models can Strategically Underperform on Evaluations [1.0485739694839669]
  Trustlocked AI systems are crucial for ensuring the safety of AI systems. Developers of AI systems may have incentives for sandbagging evaluations. We show that capability evaluations are vulnerable to sandbagging.
  arXiv  Detail & Related papers  (2024-06-11T15:26:57Z)
• Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness [53.91018508439669]
  The study explores the complexities of integrating Artificial Intelligence into Autonomous Vehicles (AVs) It examines the challenges introduced by AI components and the impact on testing procedures. The paper identifies significant challenges and suggests future directions for research and development of AI in AV technology.
  arXiv  Detail & Related papers  (2024-02-21T08:29:42Z)
• A Case for Competent AI Systems $-$ A Concept Note [0.3626013617212666]
  This note explores the concept of capability within AI systems, representing what the system is expected to deliver. The achievement of this capability may be hindered by deficiencies in implementation and testing. A central challenge arises in elucidating the competency of an AI system to execute tasks effectively.
  arXiv  Detail & Related papers  (2023-11-28T09:21:03Z)
• Guideline for Trustworthy Artificial Intelligence -- AI Assessment Catalog [0.0]
  It is clear that AI and business models based on it can only reach their full potential if AI applications are developed according to high quality standards. The issue of the trustworthiness of AI applications is crucial and is the subject of numerous major publications. This AI assessment catalog addresses exactly this point and is intended for two target groups.
  arXiv  Detail & Related papers  (2023-06-20T08:07:18Z)
• AI Maintenance: A Robustness Perspective [91.28724422822003]
  We introduce highlighted robustness challenges in the AI lifecycle and motivate AI maintenance by making analogies to car maintenance. We propose an AI model inspection framework to detect and mitigate robustness risks. Our proposal for AI maintenance facilitates robustness assessment, status tracking, risk scanning, model hardening, and regulation throughout the AI lifecycle.
  arXiv  Detail & Related papers  (2023-01-08T15:02:38Z)
• Never trust, always verify : a roadmap for Trustworthy AI? [12.031113181911627]
  We examine trust in the context of AI-based systems to understand what it means for an AI system to be trustworthy. We suggest a trust (resp. zero-trust) model for AI and suggest a set of properties that should be satisfied to ensure the trustworthiness of AI systems.
  arXiv  Detail & Related papers  (2022-06-23T21:13:10Z)
• Certifiable Artificial Intelligence Through Data Fusion [7.103626867766158]
  This paper reviews and proposes concerns in adopting, fielding, and maintaining artificial intelligence (AI) systems. A notional use case is presented with image data fusion to support AI object recognition certifiability considering precision versus distance.
  arXiv  Detail & Related papers  (2021-11-03T03:34:19Z)
• Trustworthy AI [75.99046162669997]
  Brittleness to minor adversarial changes in the input data, ability to explain the decisions, address the bias in their training data, are some of the most prominent limitations. We propose the tutorial on Trustworthy AI to address six critical issues in enhancing user and public trust in AI systems.
  arXiv  Detail & Related papers  (2020-11-02T20:04:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.