Exploring Systems-Thinking Approaches to Loss of Control Risk
- URL: http://arxiv.org/abs/2606.13474v1
- Date: Thu, 11 Jun 2026 15:27:31 GMT
- Title: Exploring Systems-Thinking Approaches to Loss of Control Risk
- Abstract summary: Internal deployment of agentic AI systems for coding and research creates a sociotechnical control problem that extends beyond model behaviour.<n>We treat internal-deployment Loss of Control as the inability to reliably constrain, audit, reverse, or halt AI-mediated changes.<n>We ask whether established systems-safety methods can identify risks that model-level evaluations may miss.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Internal deployment of agentic AI systems for coding and research creates a sociotechnical control problem that extends beyond model behaviour. We treat internal-deployment Loss of Control as the inability to reliably constrain, audit, reverse, or halt AI-mediated changes to code, infrastructure, evaluation, or deployment processes in time to prevent serious organisational or societal harms. We ask whether established systems-safety methods can identify risks that model-level evaluations may miss. Using a generic frontier-lab coding-agent scenario reconstructed from public materials, we apply STECA, STPA, and FRAM. The analyses surface complementary findings: published frameworks can leave governance responsibilities and feedback loops externally unverifiable; delays in monitoring and intervention can make otherwise valid control actions ineffective; and routine operational variability can gradually erode the calibration and independence of safeguards. We argue that frontier-AI risk management should pair model-focused evaluations with systems-level hazard analysis and operational assurance that tracks whether controls remain effective over time.
Related papers
- Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels [53.63220028820581]
Embodied intelligence integrates learned perception and decision making with real-time computation, control, and physical interaction.<n>We define trustworthy embodied intelligence as the sustained capacity to execute specified tasks reliably.
arXiv Detail & Related papers (2026-07-28T17:50:44Z) - Towards Auditing AI Systems in the Wild [3.9549290063964313]
We argue for the development of principled auditing frameworks that monitor deployed AI systems throughout their lifecycle.<n>This perspective highlights the need for uncertainty-aware monitoring methods, socio-technical specifications of audit criteria, and auditing infrastructures.
arXiv Detail & Related papers (2026-06-15T23:52:11Z) - Position: AI Safety Requires Effective Controllability [7.789894640525689]
We argue that AI safety requires controllability as a first-class objective.<n>Experiments with OpenClaw-based agents show that current alignment and guardrail mechanisms reduce risk, but often fail to provide persistent, authoritative, and enforceable runtime control.
arXiv Detail & Related papers (2026-05-26T14:53:24Z) - Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
arXiv Detail & Related papers (2026-05-17T10:26:37Z) - Action-Conditioned Risk Gating for Safety-Critical Control under Partial Observability [79.08785366532287]
We propose a lightweight risk-gated reinforcement learning approximation for risk-sensitive control under partial observability.<n>We evaluate the approach in two safety-critical partially observable domains: automated glucose regulation and safety-constrained navigation.
arXiv Detail & Related papers (2026-05-14T01:23:09Z) - STAMP/STPA informed characterization of Factors Leading to Loss of Control in AI Systems [0.0]
We set out to provide a more structured framework for discussing and characterizing loss of control.<n>We use this framework to assist those responsible for the safe operation of AI-containing socio-technical systems to identify causal factors leading to loss of control.
arXiv Detail & Related papers (2025-12-19T14:07:32Z) - How frontier AI companies could implement an internal audit function [0.0]
Internal audit could play a central role in strengthening safety governance.<n>This paper examines how an internal audit function could be designed to provide meaningful assurance for frontier AI developers.
arXiv Detail & Related papers (2025-12-16T20:36:58Z) - Making LLMs Reliable When It Matters Most: A Five-Layer Architecture for High-Stakes Decisions [51.56484100374058]
Current large language models (LLMs) excel in verifiable domains where outputs can be checked before action but prove less reliable for high-stakes strategic decisions with uncertain outcomes.<n>This gap, driven by mutually cognitive biases in both humans and artificial intelligence (AI) systems, threatens the defensibility of valuations and sustainability of investments in the sector.<n>This report describes a framework emerging from systematic qualitative assessment across 7 frontier-grade LLMs and 3 market-facing venture vignettes under time pressure.
arXiv Detail & Related papers (2025-11-10T22:24:21Z) - Systematic Hazard Analysis for Frontier AI using STPA [0.0]
frontier AI companies currently do not describe in detail any structured approach to identifying and analysing hazards.<n>A (Systems-Theoretic Process Analysis) is a systematic methodology for identifying how complex systems can become unsafe, leading to hazards.<n>We evaluateA's ability to broaden the scope, improve traceability and strengthen the robustness of safety assurance for frontier AI systems.
arXiv Detail & Related papers (2025-06-02T15:28:34Z) - Leveraging Traceability to Integrate Safety Analysis Artifacts into the
Software Development Process [51.42800587382228]
Safety assurance cases (SACs) can be challenging to maintain during system evolution.
We propose a solution that leverages software traceability to connect relevant system artifacts to safety analysis models.
We elicit design rationales for system changes to help safety stakeholders analyze the impact of system changes on safety.
arXiv Detail & Related papers (2023-07-14T16:03:27Z) - Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers.
We then present the pointwise feasibility conditions of the resulting safety controller.
We use these conditions to devise an event-triggered online data collection strategy.
arXiv Detail & Related papers (2022-08-23T05:02:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.