Bounding the Black Box: A Statistical Certification Framework for AI Risk Regulation

• URL: http://arxiv.org/abs/2604.21854v1
• Date: Thu, 23 Apr 2026 16:50:35 GMT
• Title: Bounding the Black Box: A Statistical Certification Framework for AI Risk Regulation
• Abstract summary: Governments have responded: the EU AI Act, the NIST Risk Management Framework, and the Council of Europe Convention all demand that high-risk systems demonstrate safety before deployment.<n>Beneath this regulatory consensus lies a critical vacuum: none specifies what acceptable risk'' means in quantitative terms.<n>We propose a two-stage framework that transforms AI risk regulation into engineering practice.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Artificial intelligence now decides who receives a loan, who is flagged for criminal investigation, and whether an autonomous vehicle brakes in time. Governments have responded: the EU AI Act, the NIST Risk Management Framework, and the Council of Europe Convention all demand that high-risk systems demonstrate safety before deployment. Yet beneath this regulatory consensus lies a critical vacuum: none specifies what ``acceptable risk'' means in quantitative terms, and none provides a technical method for verifying that a deployed system actually meets such a threshold. The regulatory architecture is in place; the verification instrument is not. This gap is not theoretical. As the EU AI Act moves into full enforcement, developers face mandatory conformity assessments without established methodologies for producing quantitative safety evidence - and the systems most in need of oversight are opaque statistical inference engines that resist white-box scrutiny. This paper provides the missing instrument. Drawing on the aviation certification paradigm, we propose a two-stage framework that transforms AI risk regulation into engineering practice. In Stage One, a competent authority formally fixes an acceptable failure probability $δ$ and an operational input domain $\varepsilon$ - a normative act with direct civil liability implications. In Stage Two, the RoMA and gRoMA statistical verification tools compute a definitive, auditable upper bound on the system's true failure rate, requiring no access to model internals and scaling to arbitrary architectures. We demonstrate how this certificate satisfies existing regulatory obligations, shifts accountability upstream to developers, and integrates with the legal frameworks that exist today.

Related papers

• Agent Security Meets Regulatory Reality -- A Practitioner Systematization of Autonomous-Agent Threats and Controls in Regulated Financial Systems [0.0]
  Large language model agents are entering regulated financial systems.<n>Security literature characterizing their attack surface is almost entirely laboratory-based.<n> practitioner guidance on deployment is neither peer-reviewed nor connected to a formal threat model.
  arXiv  Detail & Related papers  (2026-06-28T01:11:00Z)
• A New Framework for Cybersecurity Refusals in AI Agents [52.94784168139071]
  We present the first framework for establishing refusal boundaries in offensive security contexts.<n>We apply this framework to assess how current LLM-powered agents adhere to appropriate refusal boundaries across a range of web-based offensive security scenarios.
  arXiv  Detail & Related papers  (2026-05-31T15:39:39Z)
• Provably Secure Agent Guardrail [89.79561918065122]
  Existing defense architectures rely on empirical semantic guardrails and probabilistic large model adjudicators.<n>This paper proposes a new security paradigm for agents based on the fundamental limitations of logical reasoning.
  arXiv  Detail & Related papers  (2026-05-28T02:12:41Z)
• Making AI Compliance Evidence Machine-Readable [0.0]
  This paper proposes OSCAL as a candidate interchange format for AI governance.<n>We define 16 property extensions covering lifecycle phases, enforcement semantics, risk traceability, and risk-acceptance justification.<n>The architecture and reference implementation are open-source under Apache RM 2.0.
  arXiv  Detail & Related papers  (2026-04-15T11:51:54Z)
• CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation [68.53387633351484]
  We propose CORA (COnformal Risk-controlled GUI Agent), a post-policy, pre-action safeguarding framework that provides statistical guarantees on harmful executed actions.<n> CORA reformulates safety as selective action execution: we train a Guardian model to estimate action-conditional risk for each proposed step.<n>To rigorously evaluate this paradigm, we introduce Phone-Harm, a new benchmark of mobile safety violations with step-level harm labels.
  arXiv  Detail & Related papers  (2026-04-10T09:41:21Z)
• Trustworthy AI Posture (TAIP): A Framework for Continuous AI Assurance of Agentic Systems at Horizontal and Vertical scale [1.0896567381206714]
  Point-in-time, document-based audits cannot keep pace with non deterministic behaviour.<n>Crisis is dual-scale: vertically, governance and control obligations change faster than frameworks can operationalise them.<n>Risk-based regulation now requires organisations to demonstrate ongoing control adequacy and effectiveness.<n>This paper reframes trustworthiness as a continuously generated signal rather than a static certificate.
  arXiv  Detail & Related papers  (2026-02-15T10:49:47Z)
• Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies [57.521647436515785]
  We define frontier AI auditing as rigorous third-party verification of frontier AI developers' safety and security claims.<n>We introduce AI Assurance Levels (AAL-1 to AAL-4), ranging from time-bounded system audits to continuous, deception-resilient verification.
  arXiv  Detail & Related papers  (2026-01-16T18:44:09Z)
• AI Bill of Materials and Beyond: Systematizing Security Assurance through the AI Risk Scanning (AIRS) Framework [31.261980405052938]
  Assurance for artificial intelligence (AI) systems remains fragmented across software supply-chain security, adversarial machine learning, and governance documentation.<n>This paper introduces the AI Risk Scanning (AIRS) Framework, a threat-model-based, evidence-generating framework designed to operationalize AI assurance.
  arXiv  Detail & Related papers  (2025-11-16T16:10:38Z)
• Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering [1.6766200616088744]
  SAFE-AI Framework is a holistic approach emphasizing Safety, Auditability, Feedback, and Explainability.<n>We introduce a novel taxonomy of AI behaviors categorizing suggestive, generative, autonomous, and destructive actions to guide risk assessment and oversight.<n>This paper provides a roadmap for responsible AI integration in software engineering, aligning with emerging regulations like the EU AI Act and Canada's AIDA.
  arXiv  Detail & Related papers  (2025-08-15T22:13:54Z)
• Never Compromise to Vulnerabilities: A Comprehensive Survey on AI Governance [211.5823259429128]
  We propose a comprehensive framework integrating technical and societal dimensions, structured around three interconnected pillars: Intrinsic Security, Derivative Security, and Social Ethics.<n>We identify three core challenges: (1) the generalization gap, where defenses fail against evolving threats; (2) inadequate evaluation protocols that overlook real-world risks; and (3) fragmented regulations leading to inconsistent oversight.<n>Our framework offers actionable guidance for researchers, engineers, and policymakers to develop AI systems that are not only robust and secure but also ethically aligned and publicly trustworthy.
  arXiv  Detail & Related papers  (2025-08-12T09:42:56Z)
• AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons [62.374792825813394]
  This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability.<n>The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories.
  arXiv  Detail & Related papers  (2025-02-19T05:58:52Z)
• ACRIC: Securing Legacy Communication Networks via Authenticated Cyclic Redundancy Integrity Check [98.34702864029796]
  Recent security incidents in safety-critical industries exposed how the lack of proper message authentication enables attackers to inject malicious commands or alter system behavior.<n>These shortcomings have prompted new regulations that emphasize the pressing need to strengthen cybersecurity.<n>We introduce ACRIC, a message authentication solution to secure legacy industrial communications.
  arXiv  Detail & Related papers  (2024-11-21T18:26:05Z)
• Auction-Based Regulation for Artificial Intelligence [28.86995747151915]
  Regulators have moved slowly to pick up the safety, bias, and legal debris left in the wake of broken AI deployment.<n>We propose an auction-based regulatory mechanism that provably incentivizes devices to deploy compliant models.<n>We show that our regulatory auction boosts compliance rates by 20% and participation rates by 15% compared to baseline regulatory mechanisms.
  arXiv  Detail & Related papers  (2024-10-02T17:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.