Skill Coverage: A Test Adequacy Metric for Agent Skills

• URL: http://arxiv.org/abs/2606.20659v1
• Date: Tue, 09 Jun 2026 10:16:05 GMT
• Title: Skill Coverage: A Test Adequacy Metric for Agent Skills
• Abstract summary: Agent skills encode reusable procedural knowledge that guides large language model agents across tasks and execution contexts.<n>We introduce skill coverage, a test adequacy metric that treats the skill artifact as the object under test.<n>Applying skill coverage to SkillsBench reveals that existing benchmark executions cover only 39.90 to 43.98% of skill behavior constraints.
• Score: 17.60316723069748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent skills encode reusable procedural knowledge that guides large language model agents across tasks and execution contexts. Existing evaluations primarily assess skills through task level outcomes, yet task success alone does not reveal which parts of a skill have been exercised or which remain untested. We introduce skill coverage, a test adequacy metric that treats the skill artifact as the object under test. Our approach extracts observable skill behavior constraints from skill documents and measures whether an agent trajectory provides sufficient evidence to exercise and evaluate each constraint. Skill coverage uses a binary cover or not cover judgment, which reports whether a documented behavior has been exercised with sufficient observable evidence, without assigning an additional outcome label to the behavior. Applying skill coverage to SkillsBench reveals that existing benchmark executions cover only 39.90 to 43.98% of skill behavior constraints, suggesting that current benchmark tasks leave large portions of documented skill guidance untested. These findings show that successful task completion does not imply adequate testing of the skill artifact, highlighting skill coverage as a measure of how thoroughly agent skills are tested.

Related papers

• Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents [17.291924088593408]
  Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience.<n>In practice, induced skills may transfer unreliably and can even harm the agent that retrieves them.<n>We conduct a comprehensive and controlled study of how the way skills are induced shapes their transfer across tasks.
  arXiv  Detail & Related papers  (2026-08-20T17:12:08Z)
• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• SkillEval: Decomposing Agent Skill Quality into Interpretable Signals [27.757569384137767]
  General properties of the textttSKILL.md document play an important role in skill quality.<n>SkillEval learns an interpretable direction for each quality property from controlled positive--negative skill pairs.<n>SkillEval scores closely reflect downstream task performance, providing an early indication of whether a skill is likely to help an agent complete a task.
  arXiv  Detail & Related papers  (2026-08-07T07:25:17Z)
• Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses? [38.776828282073865]
  We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure.<n> Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations.<n>A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered.
  arXiv  Detail & Related papers  (2026-08-05T13:29:16Z)
• Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice [11.764180874774164]
  Cloud platforms increasingly deliver reusable Cloud Skills that guide AI agents in creating, configuring, monitoring, and troubleshooting cloud resources.<n>Existing Skill evaluation primarily measures whether a Skill improves task success.<n>This paper introduces Skill Test Coverage, which measures how completely a Skill's test suite covers its operational test obligations.
  arXiv  Detail & Related papers  (2026-07-24T06:27:53Z)
• SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment [10.850946672552277]
  We introduce SkillAudit, an end-to-end framework for skill-centered assessment.<n>It generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety.<n>After scanning top-ranked real-world skill packages, we found that over 7% of skills are at risky status.
  arXiv  Detail & Related papers  (2026-06-21T17:40:05Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization [99.18305770138677]
  Agent skills are reusable strategies that guide agent reasoning and action.<n>Current skill construction methods treat the problem as one-shot extraction.<n>We introduce SkillComposer, a framework that decomposes skill construction into three learnable operations.
  arXiv  Detail & Related papers  (2026-06-04T12:16:08Z)
• SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents [15.669318649782285]
  We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified protocol.<n>A generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses.<n> Experiments across a range of skill-generation methods and backbones show substantial performance variation.
  arXiv  Detail & Related papers  (2026-05-18T17:28:36Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement [66.44008181092832]
  LLM-based agent systems increasingly rely on agent skills sourced from open registries to extend their capabilities.<n>We introduce SkillAttack, a framework that verifies skill vulnerability exploitability through adversarial prompting.
  arXiv  Detail & Related papers  (2026-04-05T06:25:11Z)
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks [61.89812116484928]
  Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time.<n>We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers.<n>We show curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain.<n>Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming.
  arXiv  Detail & Related papers  (2026-02-13T07:06:06Z)
• Behavior Contrastive Learning for Unsupervised Skill Discovery [75.6190748711826]
  We propose a novel unsupervised skill discovery method through contrastive learning among behaviors. Under mild assumptions, our objective maximizes the MI between different behaviors based on the same skill. Our method implicitly increases the state entropy to obtain better state coverage.
  arXiv  Detail & Related papers  (2023-05-08T06:02:11Z)
• Design of Negative Sampling Strategies for Distantly Supervised Skill Extraction [19.43668931500507]
  We propose an end-to-end system for skill extraction, based on distant supervision through literal matching. We observe that using the ESCO taxonomy to select negative examples from related skills yields the biggest improvements. We release the benchmark dataset for research purposes to stimulate further research on the task.
  arXiv  Detail & Related papers  (2022-09-13T13:37:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.