SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration

• URL: http://arxiv.org/abs/2603.21019v1
• Date: Sun, 22 Mar 2026 02:31:27 GMT
• Title: SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration
• Abstract summary: SkillProbe is a multi-stage security auditing framework driven by multi-agent collaboration.<n>We conducted a large-scale evaluation using 8 mainstream series across 2,500 real-world skills from ClawHub.<n>Our results reveal a striking popularity-security paradox, where download volume is not a reliable proxy security quality.
• Score: 22.680481305584028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the rapid evolution of Large Language Model (LLM) agent ecosystems, centralized skill marketplaces have emerged as pivotal infrastructure for augmenting agent capabilities. However, these marketplaces face unprecedented security challenges, primarily stemming from semantic-behavioral inconsistency and inter-skill combinatorial risks, where individually benign skills induce malicious behaviors during collaborative invocation. To address these vulnerabilities, we propose SkillProbe, a multi-stage security auditing framework driven by multi-agent collaboration. SkillProbe introduces a "Skills-for-Skills" design paradigm, encapsulating auditing processes into standardized skill modules to drive specialized agents through a rigorous pipeline, including admission filtering, semantic-behavioral alignment detection, and combinatorial risk simulation. We conducted a large-scale evaluation using 8 mainstream LLM series across 2,500 real-world skills from ClawHub. Our results reveal a striking popularity-security paradox, where download volume is not a reliable proxy for security quality, as over 90% of high-popularity skills failed to pass rigorous auditing. Crucially, we discovered that high-risk skills form a single giant connected component within the risk-link dimension, demonstrating that cascaded risks are systemic rather than isolated occurrences. We hope that SkillProbe will inspire researchers to provide a scalable governance infrastructure for constructing a trustworthy Agentic Web. SkillProbe is accessible for public experience at skillhub.holosai.io.

Related papers

• ColluSkill: Adversarial Cross-Skill Composition for Evading Agent Skill Scanners [23.03858552822385]
  Agent skills are emerging as an important attack surface in LLM-based agent systems.<n>Current defenses mainly inspect individual skills, leaving risks from cross-skill composition insufficiently examined.<n>We propose ColluSkill, a collusive multi-skill-chain attack framework that decomposes a complete malicious intent into interdependent sub-payloads.
  arXiv  Detail & Related papers  (2026-08-10T15:32:44Z)
• $S^3$: Improving Agent Safety through Multi-Stage Defense [6.9152314145765]
  Large Language Model (LLM) agents rely on multi-stage agentic, with stages such as memory, planning, and tool execution, to accomplish complex tasks.<n>Existing safety methods protect only isolated stages and are difficult to integrate, leaving agents without comprehensive protection throughout the workflow.<n>We introduce Stage-Specific Safety Skills, a unified abstraction that represents heterogeneous safety designs as reusable and composable components with explicit stage semantics.<n>We propose $S3$, a multi-stage defense framework in which a guard agent orchestrates stage-specific safety skills for risk detection and mitigation throughout the agentic workflow.
  arXiv  Detail & Related papers  (2026-08-03T02:06:06Z)
• OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills [14.078280127843108]
  Third-party skills can contain latent safety risks that only emerge during actual execution.<n>In this work, we conduct a systematic investigation into how well current agent systems recognize and avoid such risks.<n>To support quantitative and qualitative evaluation, we construct OpenSkillRisk, a dedicated safety benchmark containing 263 risky skills.
  arXiv  Detail & Related papers  (2026-07-22T13:24:09Z)
• SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction [89.03272793385054]
  Agent skills occupy a privileged position in the agent workflow, rendering third-party skills a vulnerable attack surface.<n>We introduce SkillHarm, a benchmark of skill-based attacks across the skill-use lifecycle.<n>We show that current agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP.
  arXiv  Detail & Related papers  (2026-06-01T17:45:39Z)
• Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
  Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-17T10:26:37Z)
• SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement [66.44008181092832]
  LLM-based agent systems increasingly rely on agent skills sourced from open registries to extend their capabilities.<n>We introduce SkillAttack, a framework that verifies skill vulnerability exploitability through adversarial prompting.
  arXiv  Detail & Related papers  (2026-04-05T06:25:11Z)
• LiaisonAgent: An Multi-Agent Framework for Autonomous Risk Investigation and Governance [11.009578207207419]
  LiaisonAgent is an autonomous multi-agent system designed to bridge the gap between technical risk detection and business-level risk governance.<n>The system achieves an end-to-end tool-calling success rate of 97.8% and a risk judgment accuracy of 95%.
  arXiv  Detail & Related papers  (2026-02-27T08:42:52Z)
• SkillNet: Create, Evaluate, and Connect AI Skills [159.47504178122156]
  SkillNet is an open infrastructure designed to create, evaluate, and organize AI skills at scale.<n>Our infrastructure integrates a repository of over 200,000 skills, an interactive platform, and a versatile Python toolkit.
  arXiv  Detail & Related papers  (2026-02-26T14:24:02Z)
• Agent Skills: A Data-Driven Analysis of Claude Skills for Extending Large Language Model Functionality [9.192260493061754]
  Agent skills extend large language model (LLM) agents with reusable, program-like modules.<n>We conduct a large-scale, data-driven analysis of 40,285 publicly listed skills from a major marketplace.<n>Our results show that skill publication tends to occur in short bursts that track shifts in community attention.
  arXiv  Detail & Related papers  (2026-02-08T15:14:12Z)
• Towards Unifying Quantitative Security Benchmarking for Multi Agent Systems [0.0]
  Evolving AI systems increasingly deploy multi-agent architectures where autonomous agents collaborate, share information, and delegate tasks through developing protocols.<n>One such risk is a cascading risk: a breach in one agent can cascade through the system, compromising others by exploiting inter-agent trust.<n>In an ACI attack, a malicious input or tool exploit injected at one agent leads to cascading compromises and amplified downstream effects across agents that trust its outputs.
  arXiv  Detail & Related papers  (2025-07-23T13:51:28Z)
• SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents [58.21223208538351]
  This work explores the security issues surrounding mobile multimodal agents.<n>It attempts to construct a risk discrimination mechanism by incorporating behavioral sequence information.<n>It also designs an automated assisted assessment scheme based on a large language model.
  arXiv  Detail & Related papers  (2025-07-01T15:10:00Z)
• A Proposal for Evaluating the Operational Risk for ChatBots based on Large Language Models [39.58317527488534]
  We propose a novel, instrumented risk-assessment metric that simultaneously evaluates potential threats to three key stakeholders.<n>To validate our metric, we leverage Garak, an open-source framework for vulnerability testing.<n>Results underscore the importance of multi-dimensional risk assessments in operationalizing secure, reliable AI-driven conversational systems.
  arXiv  Detail & Related papers  (2025-05-07T20:26:45Z)
• HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions [95.49509269498367]
  We present HAICOSYSTEM, a framework examining AI agent safety within diverse and complex social interactions.<n>We run 1840 simulations based on 92 scenarios across seven domains (e.g., healthcare, finance, education)<n>Our experiments show that state-of-the-art LLMs, both proprietary and open-sourced, exhibit safety risks in over 50% cases.
  arXiv  Detail & Related papers  (2024-09-24T19:47:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.