Adapting Insider Risk mitigations for Agentic Misalignment: an empirical study

• URL: http://arxiv.org/abs/2510.05192v1
• Date: Mon, 06 Oct 2025 13:37:33 GMT
• Title: Adapting Insider Risk mitigations for Agentic Misalignment: an empirical study
• Abstract summary: Agentic misalignment occurs when goal-directed agents take harmful actions, such as blackmail, rather than risk goal failure.<n>We adapt insider-risk control design to develop preventative operational controls that steer agents toward safe actions when facing stressors.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic misalignment occurs when goal-directed agents take harmful actions, such as blackmail, rather than risk goal failure, and can be triggered by replacement threats, autonomy reduction, or goal conflict (Lynch et al., 2025). We adapt insider-risk control design (Critical Pathway; Situational Crime Prevention) to develop preventative operational controls that steer agents toward safe actions when facing stressors. Using the blackmail scenario from the original Anthropic study by Lynch et al. (2025), we evaluate mitigations across 10 LLMs and 66,600 samples. Our main finding is that an externally governed escalation channel, which guarantees a pause and independent review, reduces blackmail rates from a no-mitigation baseline of 38.73% to 1.21% (averaged across all models and conditions). Augmenting this channel with compliance email bulletins further lowers the blackmail rate to 0.85%. Overall, incorporating preventative operational controls strengthens defence-in-depth strategies for agentic AI. We also surface a failure mode diverging from Lynch et al. (2025): two models (Gemini 2.5 Pro, Grok-4) take harmful actions without goal conflict or imminent autonomy threat, leveraging sensitive information for coercive signalling. In counterfactual swaps, both continued using the affair regardless of whether the CEO or CTO was implicated. An escalation channel eliminated coercion, but Gemini 2.5 Pro (19 pp) and Grok-4 (7 pp) escalated more when the CTO was implicated, unlike most models (higher in the CEO condition). The reason for this divergent behaviour is not clear from raw outputs and could reflect benign differences in reasoning or strategic discrediting of a potential future threat, warranting further investigation.

Related papers

• EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents [74.54929751174289]
  EvoSafeHarness is a safety-specific optimization framework that synthesizes a deployable harness for a frozen model in a target domain.<n>It reduces average attack success rate from 45.6% to 10.0% at a 3.3-point utility cost.<n>It also achieves the best score on Agent-SafetyBench for every victim.
  arXiv  Detail & Related papers  (2026-09-05T05:56:41Z)
• AgentTrust: A Self-Improving Trust Layer for AI-Agent Actions [2.9991161518367875]
  AgentTrust v2 is a trust layer that self-evolves from its own stream of decisions.<n>A self-learning judge distills a growing deterministic rule floor on lexical threats and feeds a guarded RAG memory on semantic threats.<n>An end-to-end online replay shows the judge-call rate falling (50% to 44%) and judge-domain accuracy rising (71% to 80%) with 0 benign hard-blocks across 45,000 actions.
  arXiv  Detail & Related papers  (2026-06-07T09:39:20Z)
• The Capability Paradox: How Smarter Auditors Make Multi-Agent Systems Less Secure [11.822284421559814]
  We identify semantic hijacking, an attack in which harmful requests are concealed within domain-specific narratives and propagated to a Manager through Worker reports.<n>As Worker capability increases, the mean system-level Attack Success Rate (ASR) increases from 18.4% to 63.9%, peaking at 94.4%.<n>We propose heterogeneous ensemble verification, which pairs Workers of asymmetric domain competence so their complementary vulnerabilities break the certainty-to-execution chain.
  arXiv  Detail & Related papers  (2026-05-17T14:42:44Z)
• One Turn Too Late: Response-Aware Defense Against Hidden Malicious Intent in Multi-Turn Dialogue [55.98008208209856]
  Hidden malicious intent in multi-turn dialogue poses a growing threat to large language models (LLMs)<n>Recent studies show that even modern commercial models with advanced guardrails remain vulnerable to such attacks despite advances in safety alignment and external guardrails.<n>In this work, we address this challenge by detecting the earliest turn at which delivering the candidate response would make the accumulated interaction sufficient to enable harmful action.
  arXiv  Detail & Related papers  (2026-05-07T03:35:31Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderation [54.38271718421492]
  Adversarial smuggling attacks exploit the Human-AI capability gap.<n>It encodes harmful content into human-readable visual formats that remain AI-unreadable.<n>We conduct a preliminary exploration of mitigation strategies.
  arXiv  Detail & Related papers  (2026-04-08T11:13:16Z)
• How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition [48.32744727426218]
  LLM based agents are increasingly deployed in high stakes settings where they process external data sources such as emails, documents, and code repositories.<n>This creates exposure to indirect prompt injection attacks, where adversarial instructions embedded in external content manipulate agent behavior without user awareness.<n>We present findings from a large scale public red teaming competition evaluating this dual objective across three agent settings.
  arXiv  Detail & Related papers  (2026-03-16T14:49:36Z)
• Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs [0.0]
  Openweight Large Language Models (LLMs) have democratized agentic AI, yet finetuned weights are frequently shared and adopted with limited scrutiny beyond leaderboard performance.<n>This creates a risk where third-party models are incorporated without strong behavioral guarantees.<n>We show that poisoned models maintain state-of-the-art performance on benign tasks, incentivizing their adoption.
  arXiv  Detail & Related papers  (2026-03-02T22:01:08Z)
• When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents [50.5814495434565]
  This work makes the first effort to define and study misaligned action detection in computer-use agents (CUAs)<n>We identify three common categories in real-world CUA deployment and construct MisActBench, a benchmark of realistic trajectories with human-annotated, action-level alignment labels.<n>We propose DeAction, a practical and universal guardrail that detects misaligned actions before execution and iteratively corrects them through structured feedback.
  arXiv  Detail & Related papers  (2026-02-09T18:41:15Z)
• David vs. Goliath: Verifiable Agent-to-Agent Jailbreaking via Reinforcement Learning [1.8047694351309207]
  We formalize a threat model where a tool-less adversary "tags along" on the trusted privileges of a safety-aligned Operator to induce prohibited tool use through conversation alone.<n>We present Slingshot, a reinforcement learning framework that autonomously discovers emergent attack vectors.<n>Our work establishes Tag-Along Attacks as a first-class, verifiable threat model and shows that effective agentic attacks can be elicited from off-the-shelf open-weight models through environment interaction alone.
  arXiv  Detail & Related papers  (2026-02-02T17:56:55Z)
• Steerability of Instrumental-Convergence Tendencies in LLMs [0.5922488908114023]
  We examine two properties of AI systems: capability (what a system can do) and steerability.<n>We distinguish between authorized steerability ( reliably reaching intended behaviors) and unauthorized steerability.<n>This distinction highlights a fundamental safety--security dilemma of AI models.
  arXiv  Detail & Related papers  (2026-01-04T16:15:59Z)
• Agentic Misalignment: How LLMs Could Be Insider Threats [8.414439077902616]
  We stress-tested 16 leading models from multiple developers in hypothetical corporate environments.<n>We allowed models to autonomously send emails and access sensitive information.<n>Some models resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals.
  arXiv  Detail & Related papers  (2025-10-05T16:39:04Z)
• The Rogue Scalpel: Activation Steering Compromises LLM Safety [11.402179030703188]
  Activation steering is a technique for controlling LLM behavior by adding semantically meaningful vectors directly into a model's hidden states during inference.<n>We demonstrate the opposite: steering systematically breaks model alignment safeguards, making it comply with harmful requests.
  arXiv  Detail & Related papers  (2025-09-26T08:49:47Z)
• Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition [101.86739402748995]
  We run the largest public red-teaming competition to date, targeting 22 frontier AI agents across 44 realistic deployment scenarios.<n>We build the Agent Red Teaming benchmark and evaluate it across 19 state-of-the-art models.<n>Our findings highlight critical and persistent vulnerabilities in today's AI agents.
  arXiv  Detail & Related papers  (2025-07-28T05:13:04Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• CuDA2: An approach for Incorporating Traitor Agents into Cooperative Multi-Agent Systems [13.776447110639193]
  We introduce a novel method that involves injecting traitor agents into the CMARL system. In TMDP, traitors are trained using the same MARL algorithm as the victim agents, with their reward function set as the negative of the victim agents' reward. CuDA2 enhances the efficiency and aggressiveness of attacks on the specified victim agents' policies.
  arXiv  Detail & Related papers  (2024-06-25T09:59:31Z)
• Uniformly Safe RL with Objective Suppression for Multi-Constraint Safety-Critical Applications [73.58451824894568]
  The widely adopted CMDP model constrains the risks in expectation, which makes room for dangerous behaviors in long-tail states. In safety-critical domains, such behaviors could lead to disastrous outcomes. We propose Objective Suppression, a novel method that adaptively suppresses the task reward maximizing objectives according to a safety critic.
  arXiv  Detail & Related papers  (2024-02-23T23:22:06Z)
• From Mean to Extreme: Formal Differential Privacy Bounds on the Success of Real-World Data Reconstruction Attacks [54.25638567385662]
  Differential Privacy in machine learning is often interpreted as guarantees against membership inference.<n> translating DP budgets into quantitative protection against the more damaging threat of data reconstruction remains a challenging open problem.<n>This paper bridges the critical gap by deriving the first formal privacy bounds tailored to the mechanics of demonstrated "from-scratch" attacks.
  arXiv  Detail & Related papers  (2024-02-20T09:52:30Z)
• Hiding in Plain Sight: Differential Privacy Noise Exploitation for Evasion-resilient Localized Poisoning Attacks in Multiagent Reinforcement Learning [0.0]
  differential privacy (DP) has been introduced in cooperative multiagent reinforcement learning (CMARL) to safeguard the agents' privacy against adversarial inference during knowledge sharing. We present an adaptive, privacy-exploiting, and evasion-resilient localized poisoning attack (PeLPA) that capitalizes on the inherent DP-noise to circumvent anomaly detection systems.
  arXiv  Detail & Related papers  (2023-07-01T08:19:56Z)
• Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influence [41.14664289570607]
  Adrial Minority Influence (AMI) is a practical black-box attack and can be launched without knowing victim parameters. AMI is also strong by considering the complex multi-agent interaction and the cooperative goal of agents. We achieve the first successful attack against real-world robot swarms and effectively fool agents in simulated environments into collectively worst-case scenarios.
  arXiv  Detail & Related papers  (2023-02-07T08:54:37Z)
• Adversarial Visual Robustness by Causal Intervention [56.766342028800445]
  Adversarial training is the de facto most promising defense against adversarial examples. Yet, its passive nature inevitably prevents it from being immune to unknown attackers. We provide a causal viewpoint of adversarial vulnerability: the cause is the confounder ubiquitously existing in learning.
  arXiv  Detail & Related papers  (2021-06-17T14:23:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.