Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents
- URL: http://arxiv.org/abs/2607.12428v2
- Date: Sun, 19 Jul 2026 02:18:50 GMT
- Title: Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents
- Abstract summary: This paper presents a large-scale empirical study using the AIDev dataset to characterize security code smells in agent-generated pull requests.<n>We find that 38.9% of agent-generated PRs contain at least one security smell, with supply chain integrity issues accounting for 82.3% of all detected security smells.<n>Human collaborators are responsible for introducing 67.6% of genuine leaked secrets within these agent-assisted systems.
- Score: 3.30025498453752
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: The increasing adoption of autonomous coding agents accelerates software development but also introduces scoped security risks within high-impact file paths that can outpace traditional human review capacity. While prior research has primarily evaluated these systems in terms of functional correctness and productivity, this paper presents a large-scale empirical study using the AIDev dataset to systematically characterize security code smells in agent-generated pull requests (PRs). Through a combination of a validated LLM-as-a-judge framework and manual qualitative analysis, we identify and classify security misconfigurations across 16,112 file changes spanning 4,022 pull requests. Our results reveal that 38.9% of agent-generated PRs contain at least one security smell, with supply chain integrity issues accounting for 82.3% of all detected security smells. Furthermore, hard-coded credentials constitute 99.6% of all critical-severity security smells. Crucially, we find that human collaborators are responsible for introducing 67.6% of genuine leaked secrets within these agent-assisted workflows, while existing automated and human review processes fail to detect 81.1% of these credentials prior to integration. These findings highlight substantial security risks in agent-assisted software development workflows and suggest a potential reduction in developer vigilance. They also underscore the urgent need for context-aware security guardrails implemented directly at the point of human-AI collaboration.
Related papers
- Why Trust Your Agent? Empirical Security Gains from TRiSM-Guided Agentic Workflows in Healthcare [0.0]
This paper applies the AI Trust, Risk, and Security Management (TRiSM) framework to a medical report-generation application.<n>It demonstrates how an insecure agent workflow can be transformed into security-conscious agentic workflow.
arXiv Detail & Related papers (2026-06-27T01:00:39Z) - SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents [87.26967184869198]
SeClaw is a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents.<n> benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors.
arXiv Detail & Related papers (2026-06-01T14:23:42Z) - Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
arXiv Detail & Related papers (2026-05-17T10:26:37Z) - SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization [50.71047638695205]
Reasoning language models (RLMs) are increasingly used in programming.<n>Yet, even state-of-the-art RLMs frequently introduce critical security vulnerabilities in generated code.<n>We present SecPI, a fine-tuning pipeline that teaches RLMs to internalize structured security reasoning.
arXiv Detail & Related papers (2026-04-04T04:29:11Z) - SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistants [10.897758061443989]
SafeClaw-R is a framework that enforces safety as a system-level invariant over the execution graph.<n>We evaluate SafeClaw-R across three domains: productivity platforms, third-party skill ecosystems, and code execution environments.
arXiv Detail & Related papers (2026-03-28T05:03:54Z) - Towards Verifiably Safe Tool Use for LLM Agents [53.55621104327779]
Large language model (LLM)-based AI agents extend capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents.<n>LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records.<n>Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety.
arXiv Detail & Related papers (2026-01-12T21:31:38Z) - Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub [4.409447722044799]
This study aims to characterize how autonomous coding agents contribute to software security in practice.<n>We conduct a large-scale empirical analysis of agent-authored PRs using the AIDev dataset.<n>We then analyze prevalence, acceptance outcomes, and review latency across autonomous agents, programming ecosystems, and types of code changes.
arXiv Detail & Related papers (2026-01-01T21:14:11Z) - When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents [1.7587442088965226]
LLM-based coding agents are rapidly being deployed in software development, yet their safety implications remain poorly understood.<n>We conducted the first systematic safety evaluation of autonomous coding agents, analyzing over 12,000 actions across five state-of-the-art models.<n>We developed a high-precision detection system that identified four major vulnerability categories, with information exposure the most prevalent.
arXiv Detail & Related papers (2025-07-12T16:11:07Z) - OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety [58.201189860217724]
We introduce OpenAgentSafety, a comprehensive framework for evaluating agent behavior across eight critical risk categories.<n>Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms.<n>It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.
arXiv Detail & Related papers (2025-07-08T16:18:54Z) - SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks [11.861657542626219]
SEC-bench is the first fully automated benchmarking framework for evaluating large language model (LLM) agents.<n>Our framework automatically creates high-quality software vulnerability datasets with reproducible artifacts at a cost of only $0.87 per instance.<n>A comprehensive evaluation of state-of-the-art LLM code agents reveals significant performance gaps.
arXiv Detail & Related papers (2025-06-13T13:54:30Z) - Agent-SafetyBench: Evaluating the Safety of LLM Agents [72.92604341646691]
We introduce Agent-SafetyBench, a benchmark designed to evaluate the safety of large language models (LLMs)<n>Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions.<n>Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%.
arXiv Detail & Related papers (2024-12-19T02:35:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.