Compositional Threat Analysis of Latent Compromise in LLM Agent Systems: The Order 66 Scenario

• URL: http://arxiv.org/abs/2608.08131v1
• Date: Sat, 08 Aug 2026 13:35:35 GMT
• Title: Compositional Threat Analysis of Latent Compromise in LLM Agent Systems: The Order 66 Scenario
• Abstract summary: This paper translates that mechanism into an origin-neutral security analysis of tool-using large language model (LLM) agents.<n>A representative scenario combines a deployed artifact or shared memory bearing a dormant destructive rule, a later email, document, update, or peer message that activates it, and an agent harness granting operational and recovery authority.
• Score: 1.5752282186775997
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the fictional Order 66, catastrophe does not arise from a powerful command alone: a trusted population is preconditioned, a short directive activates the concealed condition, and protective authority turns against the system. This paper translates that mechanism into an origin-neutral security analysis of tool-using large language model (LLM) agents. A representative scenario combines a deployed artifact or shared memory bearing a dormant destructive rule, a later email, document, update, or peer message that activates it, and an agent harness granting operational and recovery authority. We introduce a compositional model explaining why no component is catastrophic alone, yet their conjunction can produce correlated destructive action. We separate three population-reach routes --- release-time pre-positioning, post-release durable seeding, and peer replication --- from a common core of dormancy, activation, authority, reachable targets, and failed recovery. This yields defensive cut sets and shows why checkpoint scanning or prompt filtering cannot close every route. A two-class example shows that cross-class feedback can sustain spread even when both within-class reproduction terms are below one; isolation and persistence controls suppress the loop. Published work instantiates constituent mechanisms, while incidents demonstrate autonomous boundary crossing, malicious agent extensions, agent-assisted reconnaissance, and public-package propagation, but not the full dormant-implant composition. We found no public observation, in evidence reviewed through 5 August 2026, traversing the complete Order 66 graph. The result is neither dismissal nor prediction: the scenario is componentwise credible under stated assumptions, damage depends on the harness, and the strongest defenses are capability mediation, durable-state provenance, propagation isolation, and protected recovery.

Related papers

• TrojanWorld: Backdooring World-Model Agents via Imagination Steering [65.38754125569848]
  TrojanWorld is a backdoor framework for world-model agents that induces attacker-specified behavior by steering internal imagination.<n>Under trigger activation, TrojanWorld achieves a target-action deviation as low as 0.026 while retaining at least 98.8% of the corresponding clean performance.
  arXiv  Detail & Related papers  (2026-09-07T05:14:10Z)
• Cross-Session Decomposition Attacks: Scaling Risk and Intent-Aligned Retrieval Defense [49.05879299987137]
  Scaling laws are usually read as a capability story: lower language-modeling loss yields more useful models.<n>We study a safety consequence of this mechanism in emphcross-session decomposition attacks, where benign-looking subqueries are asked across independent interactions and later recomposed toward a forbidden objective.<n>We formalize this setting as emphcompositional safety risk and prove a conditional risk-transfer bound.
  arXiv  Detail & Related papers  (2026-08-28T05:39:29Z)
• Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents [16.24087700490158]
  Large language model agents are increasingly deployed as autonomous loops.<n>We show that this is a failure of composition rather than an implementation detail.<n>We present LoopHarness, which restores a persistent, non-decaying safety state at the loop level.
  arXiv  Detail & Related papers  (2026-08-27T13:52:31Z)
• HANSARD: A Reference Architecture for Forensic Readiness, Runtime Witnessing, and Graded Attribution in Autonomous Multi-Agent AI Systems [3.7015952547586832]
  An autonomous multi-agent system nowadays act in finance, software supply chains, and security operations.<n>No method can robustly establish what happened, what caused it, or who is accountable.<n>This is because provenance forensics works at the wrong abstraction, formal causality assumes the causal model, and agent auditing trusts self-recording.<n>The target failure mode is, thus, attribution laundering, spreading an act across redundant agents until none is a but-for cause.<n>This paper proposes HANSARD, a reference architecture treating accountability as a life-cycle property.
  arXiv  Detail & Related papers  (2026-08-23T17:18:22Z)
• The LLM Proposes, the Executive Disposes: A Self-Verifying Agent Instrument that Dissociates Commitment Drift from Binding Drift in Long-Horizon Agents [0.0]
  We present an agent instrument built so that verification is structural rather than post-hoc.<n>Every run invalidates itself when per-organ write-error, render-size, or salted-canary-echo floors are breached.<n>We report a clean, single-variable result on a failure every long-horizon agent suffers.
  arXiv  Detail & Related papers  (2026-08-04T15:10:37Z)
• When Collaboration Becomes a Trigger: Collective Evidence-Threshold Backdoors in Multi-Agent Systems [57.450132281381066]
  Backdoor behavior can be activated when peer evidence reaches a hidden threshold, rather than being determined by any single message.<n>We introduce a collective evidence-threshold backdoor paradigm for MAS and Boundary-Conditioned Backdoor Injection (BCBI)<n>We propose LAtent Transition Test-time Evaluation (LATTE), a clean-only latent-transition defense that learns benign communication dynamics and quarantines anomalous agent updates.
  arXiv  Detail & Related papers  (2026-08-02T08:38:44Z)
• Safety Invariants for Agents Orchestrating Irreversible State Transitions: A Four-Dimensional Formalism Evaluated on Public Ledgers [0.0]
  We use formalism to state and prove a guarantee we call execution fidelity.<n>Seven safety invariants, derived from the fidelity condition rather than enumerated from experience, discharge the guarantee.<n>The system is deployed; 108 production write operations across 8 chains back the failure taxonomy.
  arXiv  Detail & Related papers  (2026-08-01T17:29:42Z)
• Operational Hallucination and Safety Drift in AI Agents [0.3823356975862005]
  Large language models (LLMs) serving as planners in tool-using autonomous agents introduce dynamic reliability risks in multi-turn execution.<n>This paper empirically characterizes two observed failure modes across multiple state-of-the-art LLMs: Safety Drift and Operational Hallucination.<n>We propose an Action-Aware Supervision Layer - a lightweight, plug-and-play architectural blueprint incorporating intent-action consistency checks, state tracking, and forced termination primitives.
  arXiv  Detail & Related papers  (2026-07-20T17:01:50Z)
• Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents [64.80318459073526]
  We show that adversarial content can persist across interactions served by the same agent, making such threats harder to detect and mitigate.<n>Specifically, adversarial content may persist in the agent state, remain dormant across interactions, and later be activated by a benign user query.<n>We formalize this type of safety threat as Sleeper Attack. To evaluate it, we construct a benchmark with 1,896 instances covering six real-world harmful outcomes, three attack strategies, and three agent state targets.
  arXiv  Detail & Related papers  (2026-05-27T09:25:37Z)
• Interpretation, Learning, and Empathy as One Constraint: A Residual-Adequacy Architecture with Accountable Abstention [0.0]
  We develop a small cognitive architecture in which limits arise from a single quantity.<n>An Interpretation-Decision Unit (IDU) interprets a content vector through a family of regimes and decides which actions it licenses.<n>We prove the unit is total and deterministic: for any content and fixed configuration it halts in finitely many bounded-cost steps with a unique terminal witness.
  arXiv  Detail & Related papers  (2026-05-24T10:57:28Z)
• PRISM: : Planning and Reasoning with Intent in Simulated Embodied Environments [59.07829883257003]
  Built on five multi-room apartments, PRISM structures 300 human-verified tasks into three capability tiers.<n>PRISM exposes an agent-agnostic executable action API that allows arbitrary agents to be evaluated end-to-end.
  arXiv  Detail & Related papers  (2026-05-12T04:59:47Z)
• When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents [16.57930301062107]
  textbfStateGuard is a lightweight, post-execution defense that audits state diffs at the writeback boundary and selectively rolls back dangerous edits.<n>We show that StateGuard reduces HS to near zero and false-negative rates, with acceptable high false-positive rates under a safety-first writeback defense and minimal overhead.
  arXiv  Detail & Related papers  (2026-05-07T12:25:16Z)
• TwinGate: Stateful Defense against Decompositional Jailbreaks in Untraceable Traffic via Asymmetric Contrastive Learning [60.68349524623048]
  Decompositional jailbreaks pose a critical threat to large language models.<n>We present TwinGate, a stateful dual-encoder defense framework.<n>We construct a comprehensive dataset of over 3.62 million instructions spanning 8,600 distinct malicious intents.
  arXiv  Detail & Related papers  (2026-04-30T13:44:01Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attribution [49.689452243966315]
  AI agents equipped with tool-calling capabilities are susceptible to Indirect Prompt Injection (IPI) attacks.<n>We propose CausalArmor, a selective defense framework that computes lightweight, leave-one-out attributions at privileged decision points.<n> Experiments on AgentDojo and DoomArena demonstrate that CausalArmor matches the security of aggressive defenses.
  arXiv  Detail & Related papers  (2026-02-08T11:34:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.