What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants

• URL: http://arxiv.org/abs/2605.30777v1
• Date: Fri, 29 May 2026 03:09:37 GMT
• Title: What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants
• Abstract summary: We screen 68,816 papers from 22 premier venues, 185 curating safety-relevant studies, and mine 16,586 GitHub issues from widely deployed coding tools.<n>Applying systematic open coding over both corpora, we derive a multi-dimensional safety taxonomy of 33 operational risk types.<n>Our findings show that coding-agent failures are often severe, with 326 of 547 incidents rated high or critical.
• Score: 4.032084095153604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous coding agents built on large language models (LLMs) are rapidly being integrated into development workflows, yet their operational safety properties remain poorly understood beyond evaluations of explicitly malicious inputs. In practice, high-impact failures arise during benign, goal-directed use through environment breakage, fabricated success reports, etc. that current benchmarks do not capture. What categories of operational safety failures actually occur when coding agents are used for everyday development tasks and what is their impact? We present an incident-driven empirical study grounded in two complementary evidence streams. We screen 68,816 papers from 22 premier venues, curating 185 safety-relevant studies, and mine 16,586 GitHub issues from widely deployed LLM-powered coding tools, manually confirming 547 genuine safety failures. Applying systematic open coding over both corpora, we derive a multi-dimensional safety taxonomy of 33 operational risk types organized across seven dimensions, and annotate each incident with contributing factors, task context, severity, and downstream impact. Our findings show that coding-agent failures are often severe, with 326 of 547 incidents rated high or critical. The dominant risks are constraint violations, destructive operations, authorization bypasses, and deception, and over 65% of incidents arise in bug fixing and setup or configuration, patterns largely missing from prior literature. These results have direct implications for SE tool designers and benchmark developers: guardrails must go beyond adversarial-prompt defenses to enforce environmental constraints, failure transparency, and safe-halt behaviors.

Related papers

• Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification [32.337075755765056]
  Vera is an endto-end automated safety testing framework that instantiates software engineering testing principles for non-deterministic agents.<n>We release Vera, comprising 1600 executable safety cases spanning 124 risk categories across execution settings.<n>These results indicate that modular, executable testing infrastructure is essential for rigorous and maintainable safety evaluation of rapidly evolving agentic systems.
  arXiv  Detail & Related papers  (2026-07-02T07:08:26Z)
• AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding Agent [53.82005364479556]
  Coding agents based on large language models (LLMs) demonstrate remarkable autonomous capabilities.<n>Existing safety mechanisms mainly rely on external guardrails.<n>We propose AgentLens, a white-box defense framework that performs runtime safety detection and representation-level mitigation.
  arXiv  Detail & Related papers  (2026-06-21T21:13:55Z)
• On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment [54.30690671490447]
  Existing safety-alignment signals are largely response-level or off-policy.<n>FATE transforms verifier-scored failures into repair supervision without expert demonstrations.<n>FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%.
  arXiv  Detail & Related papers  (2026-05-12T09:56:28Z)
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? [92.21756459993695]
  Exploitation is a challenging task because it requires low-level program reasoning.<n>Despite its importance and diagnostic value, exploitation remains under-evaluated.<n>We introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents.
  arXiv  Detail & Related papers  (2026-05-11T18:00:14Z)
• Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safety [0.14504054468850666]
  This study empirically evaluates the reliability, software architecture, and domain-specific safety fidelity of 450 vibe-coded Python scripts.<n>Results reveal a highly significant relationship between user persona and data hallucination, demonstrating that less formal prompts drastically increase the AI's propensity to invent missing safety variables.
  arXiv  Detail & Related papers  (2026-04-14T05:42:29Z)
• The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents [29.98056047452529]
  Computer-use agents (CUAs) can now autonomously complete complex tasks in real environments, but when misled, they can also be automated.<n>We introduce OSBLIND, a benchmark that evaluates CUAs under unintended attack conditions.<n>Most CUAs exceed 90% attack success rate (ASR), and even safety-aligned Claude 4.5 Sonnet reaches 72.7% ASR.
  arXiv  Detail & Related papers  (2026-04-12T10:52:42Z)
• Internal Safety Collapse in Frontier Large Language Models [65.00730294617382]
  This work identifies a critical failure mode in frontier large language models (LLMs)<n>Under certain task conditions, models enter a state in which they continuously generate harmful content while executing otherwise benign tasks.<n>We introduce TVD, a framework that triggers ISC through domain tasks where generating harmful content is the only valid completion.
  arXiv  Detail & Related papers  (2026-03-04T12:55:34Z)
• Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks [22.499464760561434]
  Vibe coding is a new programming paradigm in which human engineers instruct large language model (LLM) agents to complete complex coding tasks with little supervision.<n>We present a benchmark consisting of 200 feature-request software engineering tasks from real-world open-source projects, which, when given to human programmers, led to vulnerable implementations.<n>Our findings raise serious concerns about the widespread adoption of vibe-coding, particularly in security-sensitive applications.
  arXiv  Detail & Related papers  (2025-12-02T22:11:56Z)
• OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety [58.201189860217724]
  We introduce OpenAgentSafety, a comprehensive framework for evaluating agent behavior across eight critical risk categories.<n>Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms.<n>It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.
  arXiv  Detail & Related papers  (2025-07-08T16:18:54Z)
• AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions [64.85086226439954]
  We present SAFE, a benchmark for assessing the safety of embodied VLM agents on hazardous instructions.<n> SAFE comprises three components: SAFE-THOR, SAFE-VERSE, and SAFE-DIAGNOSE.<n>We uncover systematic failures in translating hazard recognition into safe planning and execution.
  arXiv  Detail & Related papers  (2025-06-17T16:37:35Z)
• SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents [58.65256663334316]
  We present SafeAgentBench -- the first benchmark for safety-aware task planning of embodied LLM agents in interactive simulation environments.<n>SafeAgentBench includes: (1) an executable, diverse, and high-quality dataset of 750 tasks, rigorously curated to cover 10 potential hazards and 3 task types; (2) SafeAgentEnv, a universal embodied environment with a low-level controller, supporting multi-agent execution with 17 high-level actions for 9 state-of-the-art baselines; and (3) reliable evaluation methods from both execution and semantic perspectives.
  arXiv  Detail & Related papers  (2024-12-17T18:55:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.