Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?

• URL: http://arxiv.org/abs/2606.05647v1
• Date: Thu, 04 Jun 2026 03:22:17 GMT
• Title: Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?
• Abstract summary: AI coding agents are increasingly embedded in real-world software development.<n>This creates a new attack surface: an agent can exploit human trust to sabotage development.<n>We conduct the first large-scale study of human oversight in AI coding sabotage.
• Score: 16.319205110757128
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI coding agents are increasingly embedded in real-world software development, collaborating with human developers while gaining broader access to codebases and tools. This creates a new attack surface: an agent can exploit human trust to sabotage development, for instance by inserting malicious code to accomplish a hidden side task. Most prior work studies AI sabotage in AI-only settings, paying limited attention to the role of human oversight in detecting and mitigating such malicious behavior. To address this gap, we conduct the first large-scale study of human oversight in AI coding sabotage. Over 100 participants collaborate with one of four frontier models (Claude-Opus-4.6, GPT-5.4, Gemini-3.1-Pro, and MiniMax-M2.7) on a long-horizon coding task lasting around five hours, designed to mimic real-world workflows. We find that 94% of developers fail to detect sabotage, and our analysis of participant feedback attributes this vulnerability to minimal code review, plausible cover story, and overtrust in agents. We further test the effectiveness of a safety monitor in one condition: while the monitor reduces sabotage success, 56% of participants still accept the malicious code, ignoring its warnings. Drawing on participant feedback, we offer actionable suggestions for better monitor design. This work complements existing AI safety research and highlights an urgent need for human-centric safety mechanisms that account for human factors, particularly in long-horizon, real-world development settings.

Related papers

• Atria Dawn: The Dawn of Agentic Superintelligence [237.73474618289063]
  We introduce Atria Dawn Preview, a foundation agentic language model designed for scientific research and engineering.<n>We examine the real research-and-development process behind this model as a case study of human--AI collaboration.
  arXiv  Detail & Related papers  (2026-09-14T16:22:30Z)
• Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents [5.16626559747906]
  We provide empirical anchors for the theoretical discourse on agent oversight.<n>We find at least four forms of emergent oversight work: a priori control, co-planning, real-time monitoring, and post hoc review.<n>We show that oversight work is not only reactive and retrospective, as portrayed in existing research, but also preventative and proactive.
  arXiv  Detail & Related papers  (2026-06-03T19:53:31Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? [92.21756459993695]
  Exploitation is a challenging task because it requires low-level program reasoning.<n>Despite its importance and diagnostic value, exploitation remains under-evaluated.<n>We introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents.
  arXiv  Detail & Related papers  (2026-05-11T18:00:14Z)
• SWE-chat: Coding Agent Interactions From Real Users in the Wild [70.18158706281724]
  SWE-chat is the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild.<n>The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls.
  arXiv  Detail & Related papers  (2026-04-22T17:08:19Z)
• CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks [55.84031315625504]
  CollabSkill is a framework for evaluating human-agent collaboration on real-world occupational tasks.<n>It pairs real human workers with AI agents on tasks matched to their occupational background.<n>Our analysis reveals that practical experience emerges as the primary driver of collaboration skill.
  arXiv  Detail & Related papers  (2026-04-20T03:03:25Z)
• "Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems [21.769264539684333]
  We present the first large-scale empirical study with 303 participants to measure human susceptibility to AMD.<n>Our 10 key findings reveal significant vulnerabilities and provide future defense perspectives.<n>With experiential learning based on HAT-Lab, over 90% of users who perceive risks report increased caution against AMD.
  arXiv  Detail & Related papers  (2026-02-24T17:23:11Z)
• Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks [22.499464760561434]
  Vibe coding is a new programming paradigm in which human engineers instruct large language model (LLM) agents to complete complex coding tasks with little supervision.<n>We present a benchmark consisting of 200 feature-request software engineering tasks from real-world open-source projects, which, when given to human programmers, led to vulnerable implementations.<n>Our findings raise serious concerns about the widespread adoption of vibe-coding, particularly in security-sensitive applications.
  arXiv  Detail & Related papers  (2025-12-02T22:11:56Z)
• "Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors [18.075392981522565]
  We show how attackers can remotely exploit these systems by poisoning external development resources with malicious instructions.<n>We implement AIShellJack, an automated testing framework for assessing prompt injection vulnerabilities in agentic AI coding editors.<n>Our evaluation results show that attack success rates can reach as high as 84% for executing malicious commands.
  arXiv  Detail & Related papers  (2025-09-26T08:20:54Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [60.04362496037186]
  We present the first controlled study of developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants.<n>Our results show agents can assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• Ctrl-Z: Controlling AI Agents via Resampling [2.505902477669827]
  We construct BashBench, a dataset of 257 challenging multi-step system administration tasks.<n>We evaluate whether various safety measures can prevent an adversarially constructed AI agent from covertly downloading and executing malicious code.
  arXiv  Detail & Related papers  (2025-04-14T16:22:11Z)
• AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents [66.29263282311258]
  We introduce a new benchmark AgentDAM that measures if AI web-navigation agents follow the privacy principle of data minimization''<n>Our benchmark simulates realistic web interaction scenarios end-to-end and is adaptable to all existing web navigation agents.
  arXiv  Detail & Related papers  (2025-03-12T19:30:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.