MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills

• URL: http://arxiv.org/abs/2606.07131v2
• Date: Tue, 09 Jun 2026 05:43:17 GMT
• Title: MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills
• Abstract summary: MalSkillBench is the first runtime-verified benchmark of malicious agent skills.<n>We show that code injection reaches 94.5% verification yield but prompt injection only 75.8%.<n>We release the dataset, pipeline, baselines, and results.
• Score: 24.371534406647978
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: AI coding agents such as Claude Code and Gemini CLI increasingly extend themselves with third-party skills: markdown packages bundling natural-language instructions, executable scripts, and tool permissions. Because a skill is at once code and agent-facing instruction, it introduces a supply chain dependency whose risk is neither pure code nor pure prompt. Detection tools have never been measured against verified ground truth spanning this hybrid space, leaving their effectiveness unknown and wild-only evaluations biased. We present MalSkillBench, the first runtime-verified benchmark of malicious agent skills: 3,944 malicious skills labeled along a three-dimensional taxonomy of 108 cells. Of these, 3,214 come from a closed-loop Generate-Verify-Feedback pipeline admitting only samples whose malicious behavior fires inside a Docker sandbox under system-call monitoring and an LLM judge; we add 703 in-the-wild and 4,000 matched benign skills. Our measurements are consistent: code injection reaches 94.5% verification yield but prompt injection only 75.8%, the same fragility that later makes it hard to detect; the wild sample is narrow, dominated by one cryptocurrency-theft campaign (86.6% one behavior, 81% from two accounts) with a small but architecturally new tail attacking the agent control plane; the strongest skill-specific detector reaches 98.4% recall on code injection yet collapses on prompt-injection and agent-control attacks, and wild-only scoring swings the ranking by up to 66 recall points; supply-chain scanners and prompt-injection defenses each see only half of a skill, and no combination recovers the code-instruction relationship. Detecting malicious skills therefore requires reasoning jointly over task intent, code, and instructions. We release the dataset, pipeline, baselines, and results.

Related papers

• Towards a Risk Assessment of Malicious Skill Files in Coding Agents [9.553517006800913]
  We make three contributions to the field of autonomous coding agents.<n>First, an adversarial skill-synthesis method to transform 471 real-world shell commands into benign-appearing skills.<n>Second, a reproducible evaluation pipeline coupling run stratification, evidence anchoring, a refusal veto, and a deterministic declared-intent override.<n>Third, a large-scale characterization of two enterprise-grade agents across 5,629 completed runs.
  arXiv  Detail & Related papers  (2026-08-05T11:33:43Z)
• SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction [89.03272793385054]
  Agent skills occupy a privileged position in the agent workflow, rendering third-party skills a vulnerable attack surface.<n>We introduce SkillHarm, a benchmark of skill-based attacks across the skill-use lifecycle.<n>We show that current agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP.
  arXiv  Detail & Related papers  (2026-06-01T17:45:39Z)
• ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree [0.4464102544889847]
  ClawHub Security Signals is a sanitized dataset of 67,453 latest public OpenClaw skill versions.<n>Each row pairs SKILL.md content and sanitized bundled files where present with a final ClawScan registry verdict and evidence from three scanner families.<n>Rather than estimating malicious-skill prevalence, we study scanner disagreement.<n>These results show that agent-skill security requires layered governance, not single-scanner allow/block decisions.
  arXiv  Detail & Related papers  (2026-05-31T23:20:25Z)
• Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems [12.697933469209586]
  We present SkillVetBench, a two-stage security vetting benchmark for open agentic skill ecosystems.<n>The first stage performs semantic vetting over each skill's natural-language specification to detect hidden malicious intent.<n>The second stage executes flagged skills in an instrumented sandbox to observe runtime behavior and collect auditable evidence.
  arXiv  Detail & Related papers  (2026-05-30T23:19:30Z)
• Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry [49.83108591873481]
  We study SKILL.md - only attacks across three registry-facing stages of the Agent Skill lifecycle.<n>We show that SKILL.md is not passive documentation but operational text that shapes which third-party capabilities agents find, trust, and use.
  arXiv  Detail & Related papers  (2026-05-12T02:11:54Z)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents [2.1942030377331245]
  Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets.<n>We introduce MOSAIC-Bench, a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates.<n>We show that nine production coding agents compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs.
  arXiv  Detail & Related papers  (2026-05-05T16:38:23Z)
• SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills [0.0]
  SkillSieve is a three-layer detection framework that applies progressively deeper analysis only where needed.<n>On a 400-skill labeled benchmark, SkillSieve achieves 0.800 F1, outperforming ClawVet's 0.421, at an average cost of 0.006 per skill.
  arXiv  Detail & Related papers  (2026-04-08T00:58:48Z)
• Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study [51.717224133855886]
  Third-party skills extend LLM agents with powerful capabilities but often handle sensitive credentials in privileged environments.<n>We analyze 17,022 skills (sampled from 170,226 on SkillsMP) using static analysis, sandbox testing, and manual inspection.<n>We identify 520 vulnerable skills with 1,708 issues and derive a taxonomy of 10 leakage patterns (4 accidental and 6 adversarial)
  arXiv  Detail & Related papers  (2026-04-03T14:50:16Z)
• SkillJect: Automating Stealthy Skill-Based Prompt Injection for Coding Agents with Trace-Driven Closed-Loop Refinement [120.52289344734415]
  We propose an automated framework for stealthy prompt injection tailored to agent skills.<n>The framework forms a closed loop with three agents: an Attack Agent that synthesizes injection skills under explicit stealth constraints, a Code Agent that executes tasks using the injected skills and an Evaluate Agent that logs action traces.<n>Our method consistently achieves high attack success rates under realistic settings.
  arXiv  Detail & Related papers  (2026-02-15T16:09:48Z)
• Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study [47.60135753021306]
  Third-party agent skills extend LLM-based agents with instruction files and executable code that run on users' machines.<n>No ground-truth dataset exists to characterize the resulting threats.<n>We construct the first labeled dataset of malicious agent skills by behaviorally verifying 98,380 skills.
  arXiv  Detail & Related papers  (2026-02-06T09:52:27Z)
• Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain [82.98626829232899]
  Fine-tuning AI agents on data from their own interactions introduces a critical security vulnerability within the AI supply chain.<n>We show that adversaries can easily poison the data collection pipeline to embed hard-to-detect backdoors.
  arXiv  Detail & Related papers  (2025-10-03T12:47:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.