Quantifying System-Level Harms from AI Adoption in Complex Sociotechnical Systems

• URL: http://arxiv.org/abs/2608.23906v1
• Date: Mon, 24 Aug 2026 23:20:58 GMT
• Title: Quantifying System-Level Harms from AI Adoption in Complex Sociotechnical Systems
• Abstract summary: Current AI evaluation remains model-centric, offering little insight into how observed behaviours might translate into system-level risk.<n>We propose a framework that links structured hazard analysis, component-level testing, and probabilistic system modelling.
• Score: 4.1508766371734795
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Artificial Intelligence (AI) is increasingly integrated into complex sociotechnical systems, including Critical National Infrastructure (CNI), where harms emerge from interactions between technical, human, and organisational elements. Yet current AI evaluation remains model-centric, offering little insight into how observed behaviours might translate into system-level risk. We propose a framework that links structured hazard analysis, component-level testing, and probabilistic system modelling to bridge this gap. By providing a traceable pathway from model behaviour to system-level outcomes, the framework enables practitioners to answer the "so what?" of AI failures, quantify their systemic impact, and move toward evidence-based and anticipatory governance of AI in complex systems. Applied to the UK's Real Time Gross Settlement (RTGS) system as an illustrative worked example, we derive AI-driven loss scenarios using Systems Theoretic Process Analysis (STPA) and examine adversarial manipulation of LLM-based trading as one such loss scenario. Component-level experiments show that simple adversarial inputs induce measurable behavioural shifts where AI recommendations are followed. Under the component-to-system mapping used here for a financial contagion model, these shifts alter system resilience, increasing bank failures and lowering the threshold at which shocks lead to cascading disruption, particularly under widespread or monopolistic AI adoption.

Related papers

• Windows Malware Detector as a Compound AI System: Trade-Offs in Accuracy, Efficiency, and Adversarial Robustness [11.831677944302614]
  Industrial Windows malware detectors are commonly described as Compound AI Systems composed of multiple heterogeneous components.<n>We propose a novel methodology that balances the trade-off among detection performance, computational requirements, and robustness.<n>We also introduce system-level threat models that capture how attackers exploit different degrees of knowledge to evade the entire Compound AI System.
  arXiv  Detail & Related papers  (2026-09-08T08:04:51Z)
• Provably Secure Agent Guardrail [89.79561918065122]
  Existing defense architectures rely on empirical semantic guardrails and probabilistic large model adjudicators.<n>This paper proposes a new security paradigm for agents based on the fundamental limitations of logical reasoning.
  arXiv  Detail & Related papers  (2026-05-28T02:12:41Z)
• Pitfalls in Evaluating Interpretability Agents [91.49742416116635]
  We build an agentic system in which a research agent iteratively designs experiments and refines hypotheses.<n>Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems.
  arXiv  Detail & Related papers  (2026-03-20T16:27:17Z)
• A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms [49.66022971508878]
  We argue that reasoning should be elevated from a modular component to the system's cognitive core.<n>We derive and systematize seven core reasoning challenges, such as the responsiveness-reasoning trade-off and social-game reasoning.<n>We identify a fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based reasoning and the millisecond-scale, safety-critical demands of vehicle control.
  arXiv  Detail & Related papers  (2026-03-11T07:40:53Z)
• Multi-Agent Collaborative Intrusion Detection for Low-Altitude Economy IoT: An LLM-Enhanced Agentic AI Framework [60.72591149679355]
  The rapid expansion of low-altitude economy Internet of Things (LAE-IoT) networks has created unprecedented security challenges.<n>Traditional intrusion detection systems fail to tackle the unique characteristics of aerial IoT environments.<n>We introduce a large language model (LLM)-enabled agentic AI framework for enhancing intrusion detection in LAE-IoT networks.
  arXiv  Detail & Related papers  (2026-01-25T12:47:25Z)
• Toward Quantitative Modeling of Cybersecurity Risks Due to AI Misuse [50.87630846876635]
  We develop nine detailed cyber risk models.<n>Each model decomposes attacks into steps using the MITRE ATT&CK framework.<n>Individual estimates are aggregated through Monte Carlo simulation.
  arXiv  Detail & Related papers  (2025-12-09T17:54:17Z)
• The Missing Variable: Socio-Technical Alignment in Risk Evaluation [5.539407031861404]
  Existing risk evaluation methods fail to account for the associated complex interaction between human, technical, and organizational elements.<n>We introduce a novel socio-technical alignment $STA$ variable designed to be integrated into the foundational risk equation.
  arXiv  Detail & Related papers  (2025-12-06T08:59:14Z)
• AI Deception: Risks, Dynamics, and Controls [153.71048309527225]
  This project provides a comprehensive and up-to-date overview of the AI deception field.<n>We identify a formal definition of AI deception, grounded in signaling theory from studies of animal deception.<n>We organize the landscape of AI deception research as a deception cycle, consisting of two key components: deception emergence and deception treatment.
  arXiv  Detail & Related papers  (2025-11-27T16:56:04Z)
• CORTEX: Composite Overlay for Risk Tiering and Exposure in Operational AI Systems [0.812761334568906]
  This paper introduces CORTEX, a multi-layered risk scoring framework to assess and score AI system vulnerabilities.<n>It was developed on empirical analysis of over 1,200 incidents documented in the AI Incident Database (AIID)<n>The resulting composite score can be operationalized across AI risk registers, model audits, conformity checks, and dynamic governance dashboards.
  arXiv  Detail & Related papers  (2025-08-24T07:30:25Z)
• From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems [2.226040060318401]
  We translate System Theoretic Process Analysis (STPA) for analyzing AI operation and development processes. We focus on systems that rely on machine learning algorithms and conductedA on three case studies. We find that key concepts and steps of conducting anA readily apply, albeit with a few adaptations tailored for AI systems.
  arXiv  Detail & Related papers  (2024-10-29T20:43:18Z)
• Beyond Accidents and Misuse: Decoding the Structural Risk Dynamics of Artificial Intelligence [0.0]
  This paper advances the concept of structural risk by introducing a framework grounded in complex systems research.<n>We classify structural risks into three categories: antecedent structural causes, antecedent AI system causes, and deleterious feedback loops.<n>To anticipate and govern these dynamics, the paper proposes a methodological agenda incorporating scenario mapping, simulation, and exploratory foresight.
  arXiv  Detail & Related papers  (2024-06-21T05:44:50Z)
• Machine learning dismantling and early-warning signals of disintegration in complex systems [0.0]
  We show that a machine trained to dismantle relatively small systems is able to identify higher-order topological patterns. Remarkably, the machine assesses the probability that next attacks will disintegrate the system.
  arXiv  Detail & Related papers  (2021-01-07T09:39:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.