The Controllability Trap: A Governance Framework for Military AI Agents

• URL: http://arxiv.org/abs/2603.03515v1
• Date: Tue, 03 Mar 2026 20:48:01 GMT
• Title: The Controllability Trap: A Governance Framework for Military AI Agents
• Abstract summary: We propose the Agentic Military AI Governance Framework (AMAGF)<n>AMAGF is a measurable architecture structured around three pillars: Preventive Governance, Detective Governance, and Corrective Governance.<n>Its core mechanism, the Control Quality Score (CQS), is a composite real-time metric quantifying human control and enabling graduated responses as control weakens.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic AI systems - capable of goal interpretation, world modeling, planning, tool use, long-horizon operation, and autonomous coordination - introduce distinct control failures not addressed by existing safety frameworks. We identify six agentic governance failures tied to these capabilities and show how they erode meaningful human control in military settings. We propose the Agentic Military AI Governance Framework (AMAGF), a measurable architecture structured around three pillars: Preventive Governance (reducing failure likelihood), Detective Governance (real-time detection of control degradation), and Corrective Governance (restoring or safely degrading operations). Its core mechanism, the Control Quality Score (CQS), is a composite real-time metric quantifying human control and enabling graduated responses as control weakens. For each failure type, we define concrete mechanisms, assign responsibilities across five institutional actors, and formalize evaluation metrics. A worked operational scenario illustrates implementation, and we situate the framework within established agent safety literature. We argue that governance must move from a binary conception of control to a continuous model in which control quality is actively measured and managed throughout the operational lifecycle.

Related papers

• Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels [53.63220028820581]
  Embodied intelligence integrates learned perception and decision making with real-time computation, control, and physical interaction.<n>We define trustworthy embodied intelligence as the sustained capacity to execute specified tasks reliably.
  arXiv  Detail & Related papers  (2026-07-28T17:50:44Z)
• Neuro-Symbolic Agents for Regulated Process Automation: Challenges and Research Agenda [43.328395470827665]
  We argue that symbolic structures embedded in regulated industries should be treated as core architectural components.<n>We propose compliance-by-construction as a complementary paradigm to guardrail-based monitoring.<n>We call on the neuro-symbolic community to engage with regulated process automation as a high impact research domain.
  arXiv  Detail & Related papers  (2026-06-11T14:34:11Z)
• From Assistance to Agency: Rethinking Autonomy and Control in CI/CD Pipelines [0.0]
  This paper presents a vision of agentic Continuous Integration and Continuous Deployment (CI/CD) in which the central challenge is not improving task performance but designing authority transfer.<n> Drawing on research prototypes and industrial platforms, we show that current systems operate mainly at the data plane under bounded autonomy.<n>We identify three recurring patterns: constrained autonomy as the dominant design, external governance as the primary safety mechanism, and a widening gap between deployment momentum and evaluation methodology.
  arXiv  Detail & Related papers  (2026-05-08T00:15:43Z)
• EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems [14.412476605788482]
  EmbodiedGovBench is a benchmark for governance-oriented evaluation of embodied agent systems.<n>It evaluates whether the system remains controllable, policy-bounded, recoverable, auditable, and evolution-safe under realistic perturbations.
  arXiv  Detail & Related papers  (2026-04-13T08:34:04Z)
• From Governance Norms to Enforceable Controls: A Layered Translation Method for Runtime Guardrails in Agentic AI [0.0]
  This paper proposes a layered translation method that connects standards-derived governance objectives to four control layers.<n>It distinguishes governance objectives, technical controls, runtime guardrails, and assurance evidence.<n>The central claim is modest: standards should guide control placement across architecture, runtime policy, human escalation, and audit.
  arXiv  Detail & Related papers  (2026-04-06T22:49:28Z)
• Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks [46.32011684548212]
  AI agents are vulnerable to indirect prompt injection, in which malicious instructions embedded in untrusted data can trigger dangerous agent actions.<n>This position paper discusses our vision for system-level defenses against indirect prompt injection attacks.
  arXiv  Detail & Related papers  (2026-03-31T17:15:46Z)
• When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Making [68.12864562049957]
  Embodied robotic systems increasingly rely on large language model (LLM)-based agents to support high-level reasoning.<n>This raises a fundamental question for embodied agents: when should the agent reason, and when should it act?<n>We propose RARRL (Resource-Aware Reasoning via Reinforcement Learning), a hierarchical framework for resource-aware orchestration of embodied agents.
  arXiv  Detail & Related papers  (2026-03-17T15:38:50Z)
• AI Governance Control Stack for Operational Stability: Achieving Hardened Governance in AI Systems [0.0]
  This paper introduces the AI Governance Control Stack for Operational Stability, a layered governance architecture designed to ensure traceable and resilient AI system behavior.<n>The proposed control stack integrates six complementary governance layers: system-of-record version governance, evidence-based verification, decision-time explainability logging, telemetry monitoring, model drift detection, and governance escalation.
  arXiv  Detail & Related papers  (2026-03-12T19:01:42Z)
• SAGE-LLM: Towards Safe and Generalizable LLM Controller with Fuzzy-CBF Verification and Graph-Structured Knowledge Retrieval for UAV Decision [46.089736018739295]
  Large Language Models (LLM) lack domain-specific UAV control knowledge and formal safety assurances.<n>This paper proposes a train-free two-layer decision architecture based on LLMs, integrating high-level safety planning with low-level precise control.
  arXiv  Detail & Related papers  (2026-02-27T06:41:04Z)
• Agentic AI for Cybersecurity: A Meta-Cognitive Architecture for Governable Autonomy [0.0]
  This paper argues that cybersecurity orchestration should be reconceptualized as an agentic, multi-agent cognitive system.<n>We introduce a conceptual framework in which heterogeneous AI agents responsible for detection, hypothesis formation, contextual interpretation, explanation, and governance are coordinated through an explicit meta-cognitive judgement function.<n>Our contribution is to make this cognitive structure architecturally explicit and governable by embedding meta-cognitive judgement as a first-class system function.
  arXiv  Detail & Related papers  (2026-02-12T12:52:49Z)
• Aegis: Towards Governance, Integrity, and Security of AI Voice Agents [52.7512082818639]
  We propose Aegis, a framework for the governance, integrity, and security of voice agents.<n>We evaluate the framework through case studies in banking call centers, IT Support, and logistics.<n>We observe systematic differences across model families, with open-weight models exhibiting higher susceptibility.
  arXiv  Detail & Related papers  (2026-02-07T05:51:36Z)
• Position: Capability Control Should be a Separate Goal From Alignment [43.64120192267]
  This paper argues that capability control should be treated as a distinct goal from alignment.<n>We organize capability control mechanisms across the model lifecycle into three layers.<n>Key challenges include the dual-use nature of knowledge and compositional generalization.
  arXiv  Detail & Related papers  (2026-02-05T00:30:59Z)
• Institutional AI: A Governance Framework for Distributional AGI Safety [1.3763052684269788]
  We identify three structural problems that emerge from core properties of AI models.<n>The solution is Institutional AI, a system-level approach that treats alignment as a question of effective governance of AI agent collectives.
  arXiv  Detail & Related papers  (2026-01-15T17:08:26Z)
• CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents [60.98294016925157]
  AI agents are vulnerable to prompt injection attacks, where malicious content hijacks agent behavior to steal credentials or cause financial loss.<n>We introduce Single-Shot Planning for CUAs, where a trusted planner generates a complete execution graph with conditional branches before any observation of potentially malicious content.<n>Although this architectural isolation successfully prevents instruction injections, we show that additional measures are needed to prevent Branch Steering attacks.
  arXiv  Detail & Related papers  (2026-01-14T23:06:35Z)
• Making LLMs Reliable When It Matters Most: A Five-Layer Architecture for High-Stakes Decisions [51.56484100374058]
  Current large language models (LLMs) excel in verifiable domains where outputs can be checked before action but prove less reliable for high-stakes strategic decisions with uncertain outcomes.<n>This gap, driven by mutually cognitive biases in both humans and artificial intelligence (AI) systems, threatens the defensibility of valuations and sustainability of investments in the sector.<n>This report describes a framework emerging from systematic qualitative assessment across 7 frontier-grade LLMs and 3 market-facing venture vignettes under time pressure.
  arXiv  Detail & Related papers  (2025-11-10T22:24:21Z)
• Diagnose, Localize, Align: A Full-Stack Framework for Reliable LLM Multi-Agent Systems under Instruction Conflicts [75.20929587906228]
  Large Language Model (LLM)-powered multi-agent systems (MAS) have rapidly advanced collaborative reasoning, tool use, and role-specialized coordination in complex tasks.<n>However, reliability-critical deployment remains hindered by a systemic failure mode: hierarchical compliance under instruction conflicts.
  arXiv  Detail & Related papers  (2025-09-27T08:43:34Z)
• Never Compromise to Vulnerabilities: A Comprehensive Survey on AI Governance [211.5823259429128]
  We propose a comprehensive framework integrating technical and societal dimensions, structured around three interconnected pillars: Intrinsic Security, Derivative Security, and Social Ethics.<n>We identify three core challenges: (1) the generalization gap, where defenses fail against evolving threats; (2) inadequate evaluation protocols that overlook real-world risks; and (3) fragmented regulations leading to inconsistent oversight.<n>Our framework offers actionable guidance for researchers, engineers, and policymakers to develop AI systems that are not only robust and secure but also ethically aligned and publicly trustworthy.
  arXiv  Detail & Related papers  (2025-08-12T09:42:56Z)
• Limits of Safe AI Deployment: Differentiating Oversight and Control [0.0]
  "Human oversight" risk codifying vague or inconsistent interpretations of key concepts like oversight and control.<n>This paper undertakes a targeted critical review of literature on supervision outside of AI.<n>Control aims to prevent failures, while oversight focuses on detection, remediation, or incentives for future prevention.
  arXiv  Detail & Related papers  (2025-07-04T12:22:35Z)
• Oversight Structures for Agentic AI in Public-Sector Organizations [0.0]
  We identify five governance dimensions essential for responsible agent deployment.<n>We find that agent oversight poses intensified versions of three existing governance challenges.<n>We propose approaches that both adapt institutional structures and design agent oversight compatible with public sector constraints.
  arXiv  Detail & Related papers  (2025-06-05T09:57:15Z)
• Designing Control Barrier Function via Probabilistic Enumeration for Safe Reinforcement Learning Navigation [55.02966123945644]
  We propose a hierarchical control framework leveraging neural network verification techniques to design control barrier functions (CBFs) and policy correction mechanisms.<n>Our approach relies on probabilistic enumeration to identify unsafe regions of operation, which are then used to construct a safe CBF-based control layer.<n>These experiments demonstrate the ability of the proposed solution to correct unsafe actions while preserving efficient navigation behavior.
  arXiv  Detail & Related papers  (2025-04-30T13:47:25Z)
• How to evaluate control measures for LLM agents? A trajectory from today to superintelligence [4.027240141373031]
  We propose a framework for adapting affordances of red teams to advancing AI capabilities.<n>We show how knowledge of an agents's actual capability profile can inform proportional control evaluations.
  arXiv  Detail & Related papers  (2025-04-07T16:52:52Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• Value Functions are Control Barrier Functions: Verification of Safe Policies using Control Theory [46.85103495283037]
  We propose a new approach to apply verification methods from control theory to learned value functions. We formalize original theorems that establish links between value functions and control barrier functions. Our work marks a significant step towards a formal framework for the general, scalable, and verifiable design of RL-based control systems.
  arXiv  Detail & Related papers  (2023-06-06T21:41:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.