Open Problems in AI Incident Governance

• URL: http://arxiv.org/abs/2607.05163v1
• Date: Mon, 06 Jul 2026 14:46:49 GMT
• Title: Open Problems in AI Incident Governance
• Abstract summary: We examine existing frameworks related to incident governance by regulatory bodies.<n>We find that there is a lack of consistency within the aspects of definitions, classification, monitoring, and reporting.<n>These inconsistencies apply to the types of incident data that is collected and reported.
• Score: 0.4890817787260026
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI systems may produce failures after deployment that pre-deployment safety assessments do not anticipate. Managing these failures requires what we refer to as adequate \textit{AI incident governance}, where having good definitions, taxonomies, monitoring practices, reporting mechanisms, and incident analysis is essential. We examine existing frameworks related to AI incident governance by regulatory bodies and independent efforts, and find that while there are frameworks that describe how individual functions can be performed, there is a lack of consistency within the aspects of definitions, classification, monitoring, and reporting. These inconsistencies apply to the types of incident data that is collected and reported, the ways in which they are categorised, and as a result, the depth, representativeness, and accuracy of analysis that can be performed.

Related papers

• AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation [48.55515767840701]
  "AI slop" is artifacts, hallucinated vulnerabilities, plausible but incorrect patches, and semantically repackaged bug reports.<n>This paper surveys the empirical evidence, identifies a unifying mechanism, and traces a path toward trustworthy triage.
  arXiv  Detail & Related papers  (2026-08-26T11:48:38Z)
• Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
  This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
  arXiv  Detail & Related papers  (2026-07-29T05:16:59Z)
• Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels [53.63220028820581]
  Embodied intelligence integrates learned perception and decision making with real-time computation, control, and physical interaction.<n>We define trustworthy embodied intelligence as the sustained capacity to execute specified tasks reliably.
  arXiv  Detail & Related papers  (2026-07-28T17:50:44Z)
• Exploring Systems-Thinking Approaches to Loss of Control Risk [0.0]
  Internal deployment of agentic AI systems for coding and research creates a sociotechnical control problem that extends beyond model behaviour.<n>We treat internal-deployment Loss of Control as the inability to reliably constrain, audit, reverse, or halt AI-mediated changes.<n>We ask whether established systems-safety methods can identify risks that model-level evaluations may miss.
  arXiv  Detail & Related papers  (2026-06-11T15:27:31Z)
• Neuro-Symbolic Agents for Regulated Process Automation: Challenges and Research Agenda [43.328395470827665]
  We argue that symbolic structures embedded in regulated industries should be treated as core architectural components.<n>We propose compliance-by-construction as a complementary paradigm to guardrail-based monitoring.<n>We call on the neuro-symbolic community to engage with regulated process automation as a high impact research domain.
  arXiv  Detail & Related papers  (2026-06-11T14:34:11Z)
• Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
  Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-17T10:26:37Z)
• Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demands [4.125187280299246]
  We argue that behavioural assurance, even when carefully designed, is being asked to carry safety claims it cannot verify.<n>We formalize this structural mismatch as the audit gap, the divergence between required and achievable verification access, and introduce the concept of fragile assurance.<n>We propose a technical pivot: bounding the weight of behavioral evidence in legal text and extending voluntary pre-deployment access with mechanistic-evidence classes.
  arXiv  Detail & Related papers  (2026-05-14T17:54:28Z)
• Compliance versus Sensibility: On the Reasoning Controllability in Large Language Models [46.26628756478016]
  Large Language Models (LLMs) are known to acquire reasoning capabilities through shared inference patterns in pre-training data.<n>We show that reasoning conflicts are internally detectable, as confidence scores significantly drop during conflicting episodes.<n>Our findings establish that while LLM reasoning is anchored to concrete instances, active mechanistic interventions can effectively decouple logical schemata from data.
  arXiv  Detail & Related papers  (2026-04-29T22:55:40Z)
• I Can't Believe It's Corrupt: Evaluating Corruption in Multi-Agent Governance Systems [8.670200461690454]
  We present evidence that integrity in institutional AI should be treated as a pre-deployment requirement rather than a post-deployment assumption.<n>We evaluate multi-agent governance simulations in which agents occupy formal governmental roles under different authority structures.<n>We score rule-breaking and abuse outcomes with an independent-based judge across 28,112 transcript segments.
  arXiv  Detail & Related papers  (2026-03-19T13:34:54Z)
• QCAgent: An agentic framework for quality-controllable pathology report generation from whole slide image [44.81983055985989]
  QCAgent is an agentic framework for quality-controllable pathology report generation from whole-slide image (WSI)<n>It incorporates a customized critique mechanism guided by a user-defined checklist specifying required diagnostic details and constraints.<n>It re-identifies informative regions in the WSI based on the critique feedback and text-patch semantic retrieval.
  arXiv  Detail & Related papers  (2026-03-02T09:27:31Z)
• Computable Gap Assessment of Artificial Intelligence Governance in Children's Centres: Evidence-Mechanism-Governance-Indicator Modelling of UNICEF's Guidance on AI and Children 3.0 Based on the Graph-GAP Framework [5.260137087369841]
  We propose a methodology that decomposes requirements from authoritative policy texts into a four layer graph of evidence, mechanism, governance, and indicator.<n>Using the UNICEF Innocenti Guidance on AI and Children 3.0 as primary material, we define reproducible extraction units, coding manuals, graph patterns, scoring scales, and consistency checks.<n>Results suggest that compared with privacy and data protection, requirements related to child well being and development, explainability and accountability, and cross agency implementation and resource allocation are more prone to indicator gaps and mechanism gaps.
  arXiv  Detail & Related papers  (2025-12-20T17:03:17Z)
• The Loss of Control Playbook: Degrees, Dynamics, and Preparedness [39.39076397908963]
  This report addresses the absence of an actionable definition for Loss of Control (LoC) in AI systems by developing a novel taxonomy and preparedness framework.<n>We propose a graded LoC taxonomy, based on the metrics of severity and persistence, that distinguishes between Deviation, Bounded LoC, and Strict LoC.<n>We put forward a plan to maintain preparedness and prevent the occurrence of LoC outcomes should a state of societal vulnerability be reached.
  arXiv  Detail & Related papers  (2025-11-19T20:10:39Z)
• Limits of Safe AI Deployment: Differentiating Oversight and Control [0.0]
  "Human oversight" risk codifying vague or inconsistent interpretations of key concepts like oversight and control.<n>This paper undertakes a targeted critical review of literature on supervision outside of AI.<n>Control aims to prevent failures, while oversight focuses on detection, remediation, or incentives for future prevention.
  arXiv  Detail & Related papers  (2025-07-04T12:22:35Z)
• Context-Awareness and Interpretability of Rare Occurrences for Discovery and Formalization of Critical Failure Modes [3.140125449151061]
  Vision systems are increasingly deployed in critical domains such as surveillance, law enforcement, and transportation.<n>To address these challenges, we introduce Context-Awareness and Interpretability of Rare Occurrences (CAIRO)<n>CAIRO incentivizes human-in-the-loop for testing and evaluation of criticality that arises from misdetections, adversarial attacks, and hallucinations in AI black-box models.
  arXiv  Detail & Related papers  (2025-04-18T17:12:37Z)
• Analyzing Adversarial Inputs in Deep Reinforcement Learning [53.3760591018817]
  We present a comprehensive analysis of the characterization of adversarial inputs, through the lens of formal verification. We introduce a novel metric, the Adversarial Rate, to classify models based on their susceptibility to such perturbations. Our analysis empirically demonstrates how adversarial inputs can affect the safety of a given DRL system with respect to such perturbations.
  arXiv  Detail & Related papers  (2024-02-07T21:58:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.