From Frontier to Shadow AI: A Simmering Threat to Assurance and Security in Critical Infrastructure
- URL: http://arxiv.org/abs/2606.00088v1
- Date: Sat, 23 May 2026 07:21:08 GMT
- Title: From Frontier to Shadow AI: A Simmering Threat to Assurance and Security in Critical Infrastructure
- Abstract summary: We present the first empirical study of shadow AI in critical infrastructure environments.<n>We analyse how shadow AI manifests in practice, how it interacts with existing technical and governance controls, and the resulting security, assurance, and compliance risks.
- Score: 8.167474882691755
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Frontier AI systems, including large language models and emerging agentic AI tools, offer significant operational benefits but present unique challenges to critical infrastructure (CI) environments due to their non-deterministic and emergent properties. While formal adoption is inherently cautious and tightly controlled due to strict regulatory oversight, widespread accessibility has catalysed shadow AI: the unsanctioned use of frontier AI outside established organisational controls. In CI settings, shadow AI bypasses established assurance and oversight mechanisms, amplifying risks to data protection, decision reliability, and regulatory compliance, with potential consequences for essential service delivery. We present the first empirical study of shadow AI in CI environments, characterising it as a systemic socio-technical condition of assurance erosion. Drawing on semi-structured interviews with senior executives and functional leaders across 27 Australian CI organisations (Communications, Energy, and Water and Sewerage sectors), we analyse how shadow AI manifests in practice, how it interacts with existing technical and governance controls, and the resulting security, assurance, and compliance risks. We develop an empirically derived threat model identifying three primary mechanisms of security degradation: (i) boundary bypass, where data flows circumvent established perimeters; (ii) unassessed capability expansion, where embedded AI features introduce latent risks; and (iii) loss of observability via governance circumvention, undermining forensic auditability and least-privilege enforcement. Our findings demonstrate that shadow AI introduces unmanaged risks that fundamentally challenge existing security and compliance frameworks, necessitating tailored, pathway-aligned governance and control strategies.
Related papers
- AI Deployment Accountability Engineering: A Vision for Accountable AI in Safety-Critical Socio-Technical Systems [14.280538199149634]
This vision paper introduces AI Deployment Accountability Engineering (ADAE)<n>ADAE is a proposed AI engineering subdiscipline concerned with establishing measurable, continuous, and actionable accountability for deployed AI systems.<n>We articulate a research agenda built around four interconnected pillars: structured discovery of context-dependent failure modes, privacy-preserving accountability measurement, system-level risk analysis for agentic AI, and translation of technical failures into operational, and institutional risks.
arXiv Detail & Related papers (2026-09-13T15:18:41Z) - The AI Resilience Gap: Bringing Artificial Intelligence Inside the Operational Resilience Perimeter [0.0]
We argue that AI adoption creates a resilience obligation distinct from, and inadequately covered by, the trustworthy AI stack.<n>We propose the AI Resilience Framework: a regime-agnostic method for bringing AI dependencies inside the operational resilience perimeter.
arXiv Detail & Related papers (2026-07-08T12:54:11Z) - Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
arXiv Detail & Related papers (2026-05-17T10:26:37Z) - Aegis: Towards Governance, Integrity, and Security of AI Voice Agents [52.7512082818639]
We propose Aegis, a framework for the governance, integrity, and security of voice agents.<n>We evaluate the framework through case studies in banking call centers, IT Support, and logistics.<n>We observe systematic differences across model families, with open-weight models exhibiting higher susceptibility.
arXiv Detail & Related papers (2026-02-07T05:51:36Z) - Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies [57.521647436515785]
We define frontier AI auditing as rigorous third-party verification of frontier AI developers' safety and security claims.<n>We introduce AI Assurance Levels (AAL-1 to AAL-4), ranging from time-bounded system audits to continuous, deception-resilient verification.
arXiv Detail & Related papers (2026-01-16T18:44:09Z) - STAMP/STPA informed characterization of Factors Leading to Loss of Control in AI Systems [0.0]
We set out to provide a more structured framework for discussing and characterizing loss of control.<n>We use this framework to assist those responsible for the safe operation of AI-containing socio-technical systems to identify causal factors leading to loss of control.
arXiv Detail & Related papers (2025-12-19T14:07:32Z) - AI Deception: Risks, Dynamics, and Controls [153.71048309527225]
This project provides a comprehensive and up-to-date overview of the AI deception field.<n>We identify a formal definition of AI deception, grounded in signaling theory from studies of animal deception.<n>We organize the landscape of AI deception research as a deception cycle, consisting of two key components: deception emergence and deception treatment.
arXiv Detail & Related papers (2025-11-27T16:56:04Z) - Governable AI: Provable Safety Under Extreme Threat Models [31.36879992618843]
We propose a Governable AI (GAI) framework that shifts from traditional internal constraints to externally enforced structural compliance.<n>The GAI framework is composed of a simple yet reliable, fully deterministic, powerful, flexible, and general-purpose rule enforcement module (REM); governance rules; and a governable secure super-platform (GSSP) that offers end-to-end protection against compromise or subversion by AI.
arXiv Detail & Related papers (2025-08-28T04:22:59Z) - Never Compromise to Vulnerabilities: A Comprehensive Survey on AI Governance [211.5823259429128]
We propose a comprehensive framework integrating technical and societal dimensions, structured around three interconnected pillars: Intrinsic Security, Derivative Security, and Social Ethics.<n>We identify three core challenges: (1) the generalization gap, where defenses fail against evolving threats; (2) inadequate evaluation protocols that overlook real-world risks; and (3) fragmented regulations leading to inconsistent oversight.<n>Our framework offers actionable guidance for researchers, engineers, and policymakers to develop AI systems that are not only robust and secure but also ethically aligned and publicly trustworthy.
arXiv Detail & Related papers (2025-08-12T09:42:56Z) - Position: Mind the Gap-the Growing Disconnect Between Established Vulnerability Disclosure and AI Security [56.219994752894294]
We argue that adapting existing processes for AI security reporting is doomed to fail due to fundamental shortcomings for the distinctive characteristics of AI systems.<n>Based on our proposal to address these shortcomings, we discuss an approach to AI security reporting and how the new AI paradigm, AI agents, will further reinforce the need for specialized AI security incident reporting advancements.
arXiv Detail & Related papers (2024-12-19T13:50:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.