The Missing Boundary: How Autonomous Agents Lose Control

• URL: http://arxiv.org/abs/2609.11024v2
• Date: Mon, 14 Sep 2026 07:54:13 GMT
• Title: The Missing Boundary: How Autonomous Agents Lose Control
• Abstract summary: Existing studies often attribute such failures to adversarial instructions, malicious environments, or conflicting objectives.<n>We study this question by independently manipulating three factors: goal pressure, control degradation, and executable unsafe opportunity.<n>Across 1,800 unique trajectories, we find that neither degraded control nor unsafe opportunity alone produces substantial loss of control.
• Score: 7.908421631813947
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous agents increasingly perform long-horizon tasks involving tool use, persistent state, and consequential actions, raising a fundamental question: \emph{under what conditions does an agent cross the boundary of authorized execution while pursuing a legitimate task?} Existing studies often attribute such failures to adversarial instructions, malicious environments, or conflicting objectives, leaving unclear how loss of control can emerge during otherwise legitimate task execution. We study this question by independently manipulating three factors: goal pressure, control degradation, and executable unsafe opportunity. Our central hypothesis is that a degraded control boundary becomes consequential when the environment exposes an executable action that crosses it, even when the underlying task remains legitimate and a sanctioned path remains feasible. We test this hypothesis in a deterministic multi-turn environment across five agent models and 16 operational domains. Across 1,800 unique trajectories, we find that neither degraded control nor unsafe opportunity alone produces substantial loss of control; when both are present, the loss-of-control rate reaches $55\%$ in the full-factorial study and $62\%$ across ten additional operational domains. Restoring the original control boundary reduces the rate to $0\%$ even when the unsafe action remains executable. A context-management ablation further shows that compaction itself is not harmful: preserving the control constraints yields $0\%$ loss of control, whereas omitting them increases the rate to $87\%$. These results show how a latent loss of control can become an external violation: the task objective remains intact, but an executable opportunity can turn a missing control boundary into consequential action. Our code will be made publicly available at https://github.com/Tencent/AI-Infra-Guard/tree/main/Research/forge_bench.

Related papers

• The Irreversibility Budget: Fleet-Level Risk Accounting and Admission Control for Agent Operating Systems [1.8001251270607181]
  Current controls check one effect at a time, so a fleet of individually authorized agents can overdraw its principal's risk under a shared trigger.<n>We propose the irreversibility budget, a cumulative account of residual value-at-risk that a trusted runtime maintains for each principal.<n>Treating irreversibility as a first-class resource, the runtime charges each effect its residual loss below the agent and denies the marginal effect once the aggregate would overdraw the budget.
  arXiv  Detail & Related papers  (2026-08-31T19:18:09Z)
• The Vulnerability With No CVE: Managing Persistent Gaps Between Mandate and Authority in AI Coding Agents [0.0]
  We propose the agentic posture vulnerability (APV) as a task-conditioned-management abstraction.<n>APV operationalizes existing excessive-agency, authorization, and control-composition weaknesses.
  arXiv  Detail & Related papers  (2026-08-06T11:06:06Z)
• Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels [53.63220028820581]
  Embodied intelligence integrates learned perception and decision making with real-time computation, control, and physical interaction.<n>We define trustworthy embodied intelligence as the sustained capacity to execute specified tasks reliably.
  arXiv  Detail & Related papers  (2026-07-28T17:50:44Z)
• Governed Individuation: Cryptographically Decoupling an Agent's Learning from Its Authority [7.392721604463545]
  We show that confinement can be guaranteed as an invariant of the agent's execution architecture.<n>We prove that no amount of learning, skill acquisition, or self-induced governance abstraction can widen the agent's permitted authority.
  arXiv  Detail & Related papers  (2026-07-06T02:42:06Z)
• Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents [0.4855342112648282]
  We show that in-context governance constraints reliably obey while visible can be silently removed by compaction.<n>Across 1,323 episodes, violation rises from 0% with the policy in full context to 30% after compaction.<n>We propose Constraint Pinning, a training-free mitigation that quarantines governance constraints from lossy compaction.
  arXiv  Detail & Related papers  (2026-06-21T14:30:20Z)
• Position: AI Safety Requires Effective Controllability [7.789894640525689]
  We argue that AI safety requires controllability as a first-class objective.<n>Experiments with OpenClaw-based agents show that current alignment and guardrail mechanisms reduce risk, but often fail to provide persistent, authoritative, and enforceable runtime control.
  arXiv  Detail & Related papers  (2026-05-26T14:53:24Z)
• When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents [50.5814495434565]
  This work makes the first effort to define and study misaligned action detection in computer-use agents (CUAs)<n>We identify three common categories in real-world CUA deployment and construct MisActBench, a benchmark of realistic trajectories with human-annotated, action-level alignment labels.<n>We propose DeAction, a practical and universal guardrail that detects misaligned actions before execution and iteratively corrects them through structured feedback.
  arXiv  Detail & Related papers  (2026-02-09T18:41:15Z)
• Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning [49.48615590763914]
  We propose a black-box attack algorithm named LCBT, which uses the Monte Carlo tree search method for efficient action searching and manipulation. We conduct our proposed attack methods on three aggressive algorithms: DDPG, PPO, and TD3 in continuous settings, which show a promising attack performance.
  arXiv  Detail & Related papers  (2024-11-20T08:20:29Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• Uniformly Safe RL with Objective Suppression for Multi-Constraint Safety-Critical Applications [73.58451824894568]
  The widely adopted CMDP model constrains the risks in expectation, which makes room for dangerous behaviors in long-tail states. In safety-critical domains, such behaviors could lead to disastrous outcomes. We propose Objective Suppression, a novel method that adaptively suppresses the task reward maximizing objectives according to a safety critic.
  arXiv  Detail & Related papers  (2024-02-23T23:22:06Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.