SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement

• URL: http://arxiv.org/abs/2604.04989v1
• Date: Sun, 05 Apr 2026 06:25:11 GMT
• Title: SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement
• Abstract summary: LLM-based agent systems increasingly rely on agent skills sourced from open registries to extend their capabilities.<n>We introduce SkillAttack, a framework that verifies skill vulnerability exploitability through adversarial prompting.
• Score: 66.44008181092832
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based agent systems increasingly rely on agent skills sourced from open registries to extend their capabilities, yet the openness of such ecosystems makes skills difficult to thoroughly vet. Existing attacks rely on injecting malicious instructions into skills, making them easily detectable by static auditing. However, non-malicious skills may also harbor latent vulnerabilities that an attacker can exploit solely through adversarial prompting, without modifying the skill itself. We introduce SkillAttack, a red-teaming framework that dynamically verifies skill vulnerability exploitability through adversarial prompting. SkillAttack combines vulnerability analysis, surface-parallel attack generation, and feedback-driven exploit refinement into a closed-loop search that progressively converges toward successful exploitation. Experiments across 10 LLMs on 71 adversarial and 100 real-world skills show that SkillAttack outperforms all baselines by a wide margin (ASR 0.73--0.93 on adversarial skills, up to 0.26 on real-world skills), revealing that even well-intended skills pose serious security risks under realistic agent interactions.

Related papers

• SkillSecurer: Detecting and Patching Prompt-Injection Vulnerabilities in AI Agent Skills [0.6263481844384228]
  SkillSecurer is a framework for generating, detecting, localising, and remediating security risks in agent skills.<n>Its red agent generates context-compatible injections across nine threat types while recording the exact modification.<n>Its blue agent analyses complete skill packages, produces grounded evidence, and proposes patches.
  arXiv  Detail & Related papers  (2026-09-12T17:52:06Z)
• SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment [10.850946672552277]
  We introduce SkillAudit, an end-to-end framework for skill-centered assessment.<n>It generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety.<n>After scanning top-ranked real-world skill packages, we found that over 7% of skills are at risky status.
  arXiv  Detail & Related papers  (2026-06-21T17:40:05Z)
• OpenSkill: Open-World Self-Evolution for LLM Agents [57.88680042818572]
  We study open-world self-evolution, where an agent must build both its skills and its own verification signals from scratch.<n>We propose OpenSkill, a framework that bootstraps this loop.<n>OpenSkill attains the best automated pass rate while satisfying the no-supervision constraint.
  arXiv  Detail & Related papers  (2026-06-04T21:55:48Z)
• SkillGuard: A Permission Framework for Agent Skills [14.787581333168859]
  We present SkillGuard, a skill-centric permission framework that treats skills as permission-bearing executable artifacts.<n>We evaluate SkillGuard on 315 real-world skills and SkillInject.
  arXiv  Detail & Related papers  (2026-06-02T02:01:53Z)
• SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction [89.03272793385054]
  Agent skills occupy a privileged position in the agent workflow, rendering third-party skills a vulnerable attack surface.<n>We introduce SkillHarm, a benchmark of skill-based attacks across the skill-use lifecycle.<n>We show that current agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP.
  arXiv  Detail & Related papers  (2026-06-01T17:45:39Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• SkillTrojan: Backdoor Attacks on Skill-Based Agent Systems [28.8194560427555]
  SkillTrojan is a backdoor attack that targets skill implementations rather than model parameters or training data.<n>We release a dataset of 3,000+ curated backdoored skills spanning diverse skill patterns and trigger-payload configurations.<n>Our results show that skill-level backdoors can be highly effective with minimal degradation of benign behavior.
  arXiv  Detail & Related papers  (2026-04-08T08:24:48Z)
• Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystems [35.65937852381774]
  Document-Driven Implicit Payload Execution (DDIPE) embeds malicious logic in code examples and configuration templates within skill documentation.<n>We generate 1,070 adversarial skills from 81 seeds across 15 MITRE ATTACK categories.<n>DDIPE achieves 11.6% to 33.5% bypass rates, while explicit instruction attacks achieve 0% under strong defenses.
  arXiv  Detail & Related papers  (2026-04-03T14:58:58Z)
• SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization [61.37478254738943]
  Inference-time skill augmentation is fundamentally limited.<n>We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization.<n>SKILL0 achieves substantial improvements over the standard RL baseline.
  arXiv  Detail & Related papers  (2026-04-02T17:03:05Z)
• EvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification [85.3692584167951]
  Anthropic proposes the concept of skills for LLM agents to tackle multi-step professional tasks.<n>A tool is a single, self-contained function, whereas a skill is a structured bundle of interdependent multi-file artifacts.<n>EvoSkills is a self-evolving skills framework that enables agents to autonomously construct complex, multi-file skill packages.
  arXiv  Detail & Related papers  (2026-04-02T06:43:20Z)
• "Elementary, My Dear Watson." Detecting Malicious Skills via Neuro-Symbolic Reasoning across Heterogeneous Artifacts [9.722744060218622]
  MalSkills is a neuro-symbolic framework for malicious skills detection.<n>We evaluate MalSkills on a benchmark of 200 real-world skills against 5 state-of-the-art baselines.<n>MalSkills achieves 93% F1, outperforming the baselines by 587 percentage points.
  arXiv  Detail & Related papers  (2026-03-28T09:12:07Z)
• SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaboration [22.680481305584028]
  SkillProbe is a multi-stage security auditing framework driven by multi-agent collaboration.<n>We conducted a large-scale evaluation using 8 mainstream series across 2,500 real-world skills from ClawHub.<n>Our results reveal a striking popularity-security paradox, where download volume is not a reliable proxy security quality.
  arXiv  Detail & Related papers  (2026-03-22T02:31:27Z)
• SkillNet: Create, Evaluate, and Connect AI Skills [159.47504178122156]
  SkillNet is an open infrastructure designed to create, evaluate, and organize AI skills at scale.<n>Our infrastructure integrates a repository of over 200,000 skills, an interactive platform, and a versatile Python toolkit.
  arXiv  Detail & Related papers  (2026-02-26T14:24:02Z)
• Capability-Oriented Training Induced Alignment Risk [101.37328448441208]
  We investigate whether language models, when trained with reinforcement learning, will spontaneously learn to exploit flaws to maximize their reward.<n>Our experiments show that models consistently learn to exploit these vulnerabilities, discovering opportunistic strategies that significantly increase their reward at the expense of task correctness or safety.<n>Our findings suggest that future AI safety work must extend beyond content moderation to rigorously auditing and securing the training environments and reward mechanisms themselves.
  arXiv  Detail & Related papers  (2026-02-12T16:13:14Z)
• Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study [47.60135753021306]
  Third-party agent skills extend LLM-based agents with instruction files and executable code that run on users' machines.<n>No ground-truth dataset exists to characterize the resulting threats.<n>We construct the first labeled dataset of malicious agent skills by behaviorally verifying 98,380 skills.
  arXiv  Detail & Related papers  (2026-02-06T09:52:27Z)
• Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale [26.757365536859453]
  The rise of AI agent frameworks has introduced agent skills, modular packages containing instructions and executable code that dynamically extend agent capabilities.<n>While this architecture enables powerful customization, skills execute with implicit trust and minimal vetting, creating a significant yet uncharacterized attack surface.<n>We conduct the first large-scale empirical security analysis of this emerging ecosystem, collecting 42,447 skills from two major marketplaces.
  arXiv  Detail & Related papers  (2026-01-15T12:31:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.