Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice

• URL: http://arxiv.org/abs/2607.22015v1
• Date: Fri, 24 Jul 2026 06:27:53 GMT
• Title: Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice
• Abstract summary: Cloud platforms increasingly deliver reusable Cloud Skills that guide AI agents in creating, configuring, monitoring, and troubleshooting cloud resources.<n>Existing Skill evaluation primarily measures whether a Skill improves task success.<n>This paper introduces Skill Test Coverage, which measures how completely a Skill's test suite covers its operational test obligations.
• Score: 11.764180874774164
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cloud platforms increasingly deliver reusable Cloud Skills that guide AI agents in creating, configuring, monitoring, and troubleshooting cloud resources. Workflow-oriented Skills specify multiple resource operations, user choices, validation steps, and recovery behavior. Existing Skill evaluation primarily measures whether a Skill improves task success. Passing the available testcases, however, does not reveal which behaviors specified by the Skill have never been tested. This paper introduces Skill Test Coverage, which measures how completely a Skill's test suite covers its operational test obligations. We define the coverage units, testcase coverage relation, calculation procedure, and claim boundary. Because these units and links are not explicit in natural-language Skill packages, we develop a measurement pipeline that recovers operational obligations, organizes their workflow context, maps testcases using both user prompts and initial resource states, and produces an auditable coverage report. Model-assisted candidate generation is combined with expert review to ensure that the reported coverage remains grounded in the original Skill and testcase evidence. A post-review module then translates confirmed coverage gaps into source-grounded test-improvement recommendations.

Related papers

• SkillSecurer: Detecting and Patching Prompt-Injection Vulnerabilities in AI Agent Skills [0.6263481844384228]
  SkillSecurer is a framework for generating, detecting, localising, and remediating security risks in agent skills.<n>Its red agent generates context-compatible injections across nine threat types while recording the exact modification.<n>Its blue agent analyses complete skill packages, produces grounded evidence, and proposes patches.
  arXiv  Detail & Related papers  (2026-09-12T17:52:06Z)
• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment [10.850946672552277]
  We introduce SkillAudit, an end-to-end framework for skill-centered assessment.<n>It generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety.<n>After scanning top-ranked real-world skill packages, we found that over 7% of skills are at risky status.
  arXiv  Detail & Related papers  (2026-06-21T17:40:05Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior [23.11308948097101]
  Current benchmarks rarely distinguish what a Skill says from how it is organized.<n>We study this distinction through Progressive Disclosure, where a concise root file points agents to supporting resources on demand.<n>We present SkillJuror, a framework for evaluating Skill writing paradigms.
  arXiv  Detail & Related papers  (2026-06-10T01:11:50Z)
• Skill Coverage: A Test Adequacy Metric for Agent Skills [17.60316723069748]
  Agent skills encode reusable procedural knowledge that guides large language model agents across tasks and execution contexts.<n>We introduce skill coverage, a test adequacy metric that treats the skill artifact as the object under test.<n>Applying skill coverage to SkillsBench reveals that existing benchmark executions cover only 39.90 to 43.98% of skill behavior constraints.
  arXiv  Detail & Related papers  (2026-06-09T10:16:05Z)
• What Should a Skill Remember? Quality--Cost Trade-offs in Cost-Aware Skill Rewriting for Language Model Agents [58.87681796862133]
  We study skill rewriting through an economic lens.<n>Our framework profiles skill structure and rewrites skills using information-preservation strategies.<n> Experiments on SkillsBench reveal distinct quality--cost trade-offs across strategies.
  arXiv  Detail & Related papers  (2026-06-08T12:36:51Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents [15.669318649782285]
  We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified protocol.<n>A generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses.<n> Experiments across a range of skill-generation methods and backbones show substantial performance variation.
  arXiv  Detail & Related papers  (2026-05-18T17:28:36Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks [61.89812116484928]
  Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time.<n>We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers.<n>We show curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain.<n>Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming.
  arXiv  Detail & Related papers  (2026-02-13T07:06:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.