SkillAtlas: An Attack Trace Library for Agent Skills
- URL: http://arxiv.org/abs/2609.13353v1
- Date: Fri, 11 Sep 2026 16:44:39 GMT
- Title: SkillAtlas: An Attack Trace Library for Agent Skills
- Abstract summary: We present SkillAtlas, a hosted attack trace library that converts private agent-skill security report bundles into reviewed, redacted, and searchable public cases.<n>The library contains 3,014 cases, 6,589 traces, 151,131 steps, 233 affected skills, and 8 risk categories.
- Score: 67.18648151130607
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Agent skills are reusable units for language-model agents, but their risks emerge through model decisions, user context, tool calls, and execution feedback rather than through stable signatures or a single sandbox run. Existing static, dynamic, and benchmark-style evaluations rarely preserve public evidence that can be inspected, searched, and reused. We present SkillAtlas, a hosted attack trace library that converts private agent-skill security report bundles into reviewed, redacted, and searchable public cases. The library contains 3,014 cases, 6,589 traces, 151,131 steps, 233 affected skills, and 8 risk categories; 42.5% of successful cases first become successful after a non-success initial round, and trajectory-grounded labels improve pre-execution guard accuracy to 0.770.
Related papers
- Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills [9.077159743623076]
repository-native framework for evaluating skills and product capability packages as executable agent artifacts.<n>ACES runs paired live trials with and without a target skill, grades six default runtime metrics, and reports Skill Lift.<n>Open-source implementation of the methodology is available in NVIDIA SkillEvaluator.
arXiv Detail & Related papers (2026-08-20T23:26:14Z) - Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
arXiv Detail & Related papers (2026-08-14T07:26:38Z) - ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents [23.994328225666663]
Cowork agents may complete benign tasks while disclosing protected data, manipulating unauthorized state, invocate unauthorized API.<n>We define behavioral safety and introduce ActBench, a benchmark that evaluates such behavior risk from execution trajectories rather than final responses.<n>ActBench contains 600 cases from 213 scenarios, spanning 15 risk behaviors, six execution spaces, and 48 web-service APIs.
arXiv Detail & Related papers (2026-08-10T11:45:03Z) - SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction [89.03272793385054]
Agent skills occupy a privileged position in the agent workflow, rendering third-party skills a vulnerable attack surface.<n>We introduce SkillHarm, a benchmark of skill-based attacks across the skill-use lifecycle.<n>We show that current agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP.
arXiv Detail & Related papers (2026-06-01T17:45:39Z) - Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry [49.83108591873481]
We study SKILL.md - only attacks across three registry-facing stages of the Agent Skill lifecycle.<n>We show that SKILL.md is not passive documentation but operational text that shapes which third-party capabilities agents find, trust, and use.
arXiv Detail & Related papers (2026-05-12T02:11:54Z) - Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study [51.717224133855886]
Third-party skills extend LLM agents with powerful capabilities but often handle sensitive credentials in privileged environments.<n>We analyze 17,022 skills (sampled from 170,226 on SkillsMP) using static analysis, sandbox testing, and manual inspection.<n>We identify 520 vulnerable skills with 1,708 issues and derive a taxonomy of 10 leakage patterns (4 accidental and 6 adversarial)
arXiv Detail & Related papers (2026-04-03T14:50:16Z) - Malicious Or Not: Adding Repository Context to Agent Skill Classification [3.773312847224526]
We present the largest empirical security analysis of the AI agent skill ecosystem.<n>We collect 238,180 unique skills from three major distribution platforms and GitHub.<n>Our analysis also uncovers several, by now undocumented real-world attack vectors.
arXiv Detail & Related papers (2026-03-17T14:27:35Z) - Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study [47.60135753021306]
Third-party agent skills extend LLM-based agents with instruction files and executable code that run on users' machines.<n>No ground-truth dataset exists to characterize the resulting threats.<n>We construct the first labeled dataset of malicious agent skills by behaviorally verifying 98,380 skills.
arXiv Detail & Related papers (2026-02-06T09:52:27Z) - Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale [26.757365536859453]
The rise of AI agent frameworks has introduced agent skills, modular packages containing instructions and executable code that dynamically extend agent capabilities.<n>While this architecture enables powerful customization, skills execute with implicit trust and minimal vetting, creating a significant yet uncharacterized attack surface.<n>We conduct the first large-scale empirical security analysis of this emerging ecosystem, collecting 42,447 skills from two major marketplaces.
arXiv Detail & Related papers (2026-01-15T12:31:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.