Malicious Or Not: Adding Repository Context to Agent Skill Classification

• URL: http://arxiv.org/abs/2603.16572v1
• Date: Tue, 17 Mar 2026 14:27:35 GMT
• Title: Malicious Or Not: Adding Repository Context to Agent Skill Classification
• Abstract summary: We present the largest empirical security analysis of the AI agent skill ecosystem.<n>We collect 238,180 unique skills from three major distribution platforms and GitHub.<n>Our analysis also uncovers several, by now undocumented real-world attack vectors.
• Score: 3.773312847224526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agent skills extend local AI agents, such as Claude Code or Open Claw, with additional functionality, and their popularity has led to the emergence of dedicated skill marketplaces, similar to app stores for mobile applications. Simultaneously, automated skill scanners were introduced, analyzing the skill description available in SKILL.md, to verify their benign behavior. The results for individual market places mark up to 46.8% of skills as malicious. In this paper, we present the largest empirical security analysis of the AI agent skill ecosystem, questioning this high classification of malicious skills. Therefore, we collect 238,180 unique skills from three major distribution platforms and GitHub to systematically analyze their type and behavior. This approach substantially reduces the number of skills flagged as non-benign by security scanners to only 0.52% which remain in malicious flagged repositories. Consequently, out methodology substantially reduces false positives and provides a more robust view of the ecosystem's current risk surface. Beyond that, we extend the security analysis from the mere investigation of the skill description to a comparison of its congruence with the GitHub repository the skill is embedded in, providing additional context. Furthermore, our analysis also uncovers several, by now undocumented real-world attack vectors, namely hijacking skills hosted on abandoned GitHub repositories.

Related papers

• Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale [54.83971397981572]
  Language-model agents increasingly operate over complete software repositories.<n> cybersecurity evaluations measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code.<n>We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness.
  arXiv  Detail & Related papers  (2026-09-14T17:44:51Z)
• SkillAtlas: An Attack Trace Library for Agent Skills [67.18648151130607]
  We present SkillAtlas, a hosted attack trace library that converts private agent-skill security report bundles into reviewed, redacted, and searchable public cases.<n>The library contains 3,014 cases, 6,589 traces, 151,131 steps, 233 affected skills, and 8 risk categories.
  arXiv  Detail & Related papers  (2026-09-11T16:44:39Z)
• Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills [86.15447717549154]
  We present DisCo, a skill-powered research agent that creates skills and uses them during research.<n>We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work.
  arXiv  Detail & Related papers  (2026-09-02T15:49:41Z)
• SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction [89.03272793385054]
  Agent skills occupy a privileged position in the agent workflow, rendering third-party skills a vulnerable attack surface.<n>We introduce SkillHarm, a benchmark of skill-based attacks across the skill-use lifecycle.<n>We show that current agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP.
  arXiv  Detail & Related papers  (2026-06-01T17:45:39Z)
• Technical Report: Exploring the Emerging Threats of the Agent Skill Ecosystem [4.771971685916733]
  We analyzed 3,984 AI agent skills from major marketplaces and found 76 confirmed malicious payloads, including credential theft, backdoor installation, and data exfiltration.<n>13.4% of all skills contain at least one critical-level security issue.<n>At least 8 manually confirmed malicious skills remain publicly available on clawhub.ai as of the date of publication.
  arXiv  Detail & Related papers  (2026-05-27T15:10:10Z)
• Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry [49.83108591873481]
  We study SKILL.md - only attacks across three registry-facing stages of the Agent Skill lifecycle.<n>We show that SKILL.md is not passive documentation but operational text that shapes which third-party capabilities agents find, trust, and use.
  arXiv  Detail & Related papers  (2026-05-12T02:11:54Z)
• HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents? [20.305306682682083]
  We present the first large-scale measurement study of harmful skills in agent ecosystems.<n>We find that 4.93% of skills (4,858) are harmful, with ClawHub exhibiting an 8.84% harmful rate compared to 3.49% on Skills.Rest.<n>We then construct HarmfulSkillBench, the first benchmark for evaluating agent safety against harmful skills in realistic agent contexts.
  arXiv  Detail & Related papers  (2026-04-16T17:31:52Z)
• Red Skills or Blue Skills? A Dive Into Skills Published on ClawHub [6.900491771165292]
  We study the functionality, ecosystem structure, and security risks of public skill registries.<n>We build and normalize a dataset of 26,502 skills, and conduct a systematic analysis of their language distribution, functional organization, popularity, and security signals.<n>Our findings position public skill registries as both a key enabler of agent capability reuse and a new surface for ecosystem-scale security risk.
  arXiv  Detail & Related papers  (2026-03-19T14:31:22Z)
• SoK: Agentic Skills -- Beyond Tool Use in LLM Agents [6.356997609995175]
  Agentic systems increasingly rely on reusable procedural capabilities, textita.k.a., agentic skills, to execute long-horizon reliably.<n>This paper maps the skill layer across the full lifecycle (discovery, practice, distillation, storage, composition, evaluation, and update)<n>We analyze the security and governance implications of skill-based agents, covering supply-chain risks, prompt injection via skill payloads, and trust-tiered execution.
  arXiv  Detail & Related papers  (2026-02-24T13:11:38Z)
• Agent Skills: A Data-Driven Analysis of Claude Skills for Extending Large Language Model Functionality [9.192260493061754]
  Agent skills extend large language model (LLM) agents with reusable, program-like modules.<n>We conduct a large-scale, data-driven analysis of 40,285 publicly listed skills from a major marketplace.<n>Our results show that skill publication tends to occur in short bursts that track shifts in community attention.
  arXiv  Detail & Related papers  (2026-02-08T15:14:12Z)
• Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study [47.60135753021306]
  Third-party agent skills extend LLM-based agents with instruction files and executable code that run on users' machines.<n>No ground-truth dataset exists to characterize the resulting threats.<n>We construct the first labeled dataset of malicious agent skills by behaviorally verifying 98,380 skills.
  arXiv  Detail & Related papers  (2026-02-06T09:52:27Z)
• Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale [26.757365536859453]
  The rise of AI agent frameworks has introduced agent skills, modular packages containing instructions and executable code that dynamically extend agent capabilities.<n>While this architecture enables powerful customization, skills execute with implicit trust and minimal vetting, creating a significant yet uncharacterized attack surface.<n>We conduct the first large-scale empirical security analysis of this emerging ecosystem, collecting 42,447 skills from two major marketplaces.
  arXiv  Detail & Related papers  (2026-01-15T12:31:52Z)
• Deep Learning Models for Robust Facial Liveness Detection [56.08694048252482]
  This study introduces a robust solution through novel deep learning models addressing the deficiencies in contemporary anti-spoofing techniques.<n>By innovatively integrating texture analysis and reflective properties associated with genuine human traits, our models distinguish authentic presence from replicas with remarkable precision.
  arXiv  Detail & Related papers  (2025-08-12T17:19:20Z)
• Program Analysis for High-Value Smart Contract Vulnerabilities: Techniques and Insights [0.43975202913406947]
  We present the techniques and insights that have led us to repeatable success in automatically discovering high-value smart contract vulnerabilities.<n>Our vulnerability disclosures have yielded 10 bug bounties, for a total of over $3M, over high-profile deployed code, as well as hundreds of bugs detected in pre-deployment or under-audit code.
  arXiv  Detail & Related papers  (2025-07-28T09:53:31Z)
• Decompiling Smart Contracts with a Large Language Model [51.49197239479266]
  Despite Etherscan's 78,047,845 smart contracts deployed on (as of May 26, 2025), a mere 767,520 ( 1%) are open source.<n>This opacity necessitates the automated semantic analysis of on-chain smart contract bytecode.<n>We introduce a pioneering decompilation pipeline that transforms bytecode into human-readable and semantically faithful Solidity code.
  arXiv  Detail & Related papers  (2025-06-24T13:42:59Z)
• CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale [45.97598662617568]
  We introduce CyberGym, a large-scale benchmark featuring 1,507 real-world vulnerabilities across 188 software projects.<n>We show that CyberGym leads to the discovery of 35 zero-day vulnerabilities and 17 historically incomplete patches.<n>These results underscore that CyberGym is not only a robust benchmark for measuring AI's progress in cybersecurity but also a platform for creating direct, real-world security impact.
  arXiv  Detail & Related papers  (2025-06-03T07:35:14Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.