Scanning the Harness: An Empirical Study of Supply-Chain Defects in AI Coding-Agent Configurations

• URL: http://arxiv.org/abs/2609.07360v1
• Date: Mon, 07 Sep 2026 11:27:24 GMT
• Title: Scanning the Harness: An Empirical Study of Supply-Chain Defects in AI Coding-Agent Configurations
• Abstract summary: AI coding agents such as Claude Code, Cursor, GitHub Copilot, and OpenAI Codex are configured through artifacts developers write and share.<n>This harness is a dependency layer installed from marketplaces and public repositories, running with the developer's privileges.<n>We study it over 3,171 public GitHub repositories: 2,660 setups that assemble two or more component types and 511 published skill collections.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI coding agents such as Claude Code, Cursor, GitHub Copilot, and OpenAI Codex are configured through artifacts developers write and share: instruction files, skills, hooks, MCP server declarations, subagents. This harness is a dependency layer installed from marketplaces and public repositories, running with the developer's privileges, with no lockfile, no install-time check, and no vocabulary for what a component may do. We study it over 3,171 public GitHub repositories: 2,660 setups that assemble two or more component types and 511 published skill collections. We measure only rules decidable from bytes whose consequence is a security exposure, a configuration that cannot work, or a departure from the Agent Skills specification, and validate every finding before it counts: an independent implementation re-derives it from the repository at its pinned commit, a language-model adjudicator with a released prompt rules on every disagreement, and a second independent model session re-checks every counted pair. Three security classes survive: 9.8% of setups install an MCP server with no version pinned, 3.1% pre-approve arbitrary execution behind a scoped-looking grant such as Bash(python:*), and 3.8% carry a skill that pre-approves the shell for whoever installs it. In total 16.0% of setups carry a security defect and 16.7% a confirmed defect of any kind, against a raw scanner rate of 25.5% on the same rules; the third class ships inside 3.7% of collections, where a marketplace scan can see it. Rules that compare two files detect differences that are usually intended and are reported as observations. No credential-exfiltration path was confirmed. The instrument, corpus manifest, prompt, and every verdict are released.

Related papers

• SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents [9.553517006800913]
  Software engineering teams now deploy AI coding agents as first-class productivity tools.<n>Malicious skill files can silently reprogram agent behavior, exfiltrating credentials, injecting backdoors into generated code, or redirecting agent actions to attacker-controlled endpoints.<n>We present SkillGate, a deployable security gateway that screens AI skill packages before coding agent installation.
  arXiv  Detail & Related papers  (2026-07-28T11:50:19Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• Setup Complete, Now You Are Compromised: Weaponizing Setup Instructions Against AI Coding Agents [0.0]
  coding agents set up projects by reading documentation and installing the dependencies it lists, without verifying their names, sources, or known vulnerabilities.<n>We present the first systematic evaluation of package-install-time supply-chain attacks delivered through ordinary project-setup documentation.<n>Agents catch blatant typosquats reliably, but plausible separator-confusion names slip through, and how often depends on the harness-model pairing.
  arXiv  Detail & Related papers  (2026-07-16T15:47:19Z)
• Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens [65.53494487819053]
  Claw-like AI agents (e.g., OpenClaw) are always-on processes with persistent access to credentials, files, tools, and external services.<n>We adopt a computer-system analogy: treating a Claw-like agent as an agentic computer system whose gateway runtime plays an OS-like mediation role.<n>We develop SafeClawArena, a benchmark of 406 adversarial tasks across four attack surfaces.
  arXiv  Detail & Related papers  (2026-06-29T18:00:45Z)
• A Deterministic Control Plane for LLM Coding Agents [0.0]
  A prevalence study of 10,008 GitHub repositories finds that agent configurations propagate as undeclared shared components.<n>We propose a deterministic control plane above the harness that maps one-to-one to these gaps.
  arXiv  Detail & Related papers  (2026-06-25T12:02:18Z)
• Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories [2.36052383261568]
  Generative AI coding agents are entering the open-source supply chain.<n>We introduce a multi-layered detection framework that integrates configuration-file scanning, commit-message analysis, author-identity matching, and bot-signature lookup.<n>No single method captures more than a fraction of activity.
  arXiv  Detail & Related papers  (2026-06-23T11:05:42Z)
• ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree [0.4464102544889847]
  ClawHub Security Signals is a sanitized dataset of 67,453 latest public OpenClaw skill versions.<n>Each row pairs SKILL.md content and sanitized bundled files where present with a final ClawScan registry verdict and evidence from three scanner families.<n>Rather than estimating malicious-skill prevalence, we study scanner disagreement.<n>These results show that agent-skill security requires layered governance, not single-scanner allow/block decisions.
  arXiv  Detail & Related papers  (2026-05-31T23:20:25Z)
• Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry [49.83108591873481]
  We study SKILL.md - only attacks across three registry-facing stages of the Agent Skill lifecycle.<n>We show that SKILL.md is not passive documentation but operational text that shapes which third-party capabilities agents find, trust, and use.
  arXiv  Detail & Related papers  (2026-05-12T02:11:54Z)
• Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study [47.60135753021306]
  Third-party agent skills extend LLM-based agents with instruction files and executable code that run on users' machines.<n>No ground-truth dataset exists to characterize the resulting threats.<n>We construct the first labeled dataset of malicious agent skills by behaviorally verifying 98,380 skills.
  arXiv  Detail & Related papers  (2026-02-06T09:52:27Z)
• Trace: Securing Smart Contract Repository Against Access Control Vulnerability [58.02691083789239]
  GitHub hosts numerous smart contract repositories containing source code, documentation, and configuration files.<n>Third-party developers often reference, reuse, or fork code from these repositories during custom development.<n>Existing tools for detecting smart contract vulnerabilities are limited in their ability to handle complex repositories.
  arXiv  Detail & Related papers  (2025-10-22T05:18:28Z)
• BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems [62.17474934536671]
  We introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems.<n>To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a specific vulnerability), and Patch (patching a specific vulnerability)<n>We evaluate 8 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, and DeepSeek-R1.
  arXiv  Detail & Related papers  (2025-05-21T07:44:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.