SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment

• URL: http://arxiv.org/abs/2606.22613v1
• Date: Sun, 21 Jun 2026 17:40:05 GMT
• Title: SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment
• Abstract summary: We introduce SkillAudit, an end-to-end framework for skill-centered assessment.<n>It generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety.<n>After scanning top-ranked real-world skill packages, we found that over 7% of skills are at risky status.
• Score: 10.850946672552277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.

Related papers

• Coalition-Aware Skill Reliability for Self-Evolving Agents [49.43736235959317]
  We investigate whether accumulated skills in an agent's skill bank actually make positive mechanistic contributions.<n>We show that coalition pollution, where bank-level gains conceal negative coalition-level skill contributions, and cross-domain utility reversal, where source-beneficial skills reverse their effects after transfer.<n>These findings motivate two reliability interventions: coalition-aware skill selection during skill accumulation and label-free skill masking after transfer.
  arXiv  Detail & Related papers  (2026-08-23T21:35:22Z)
• SkillEval: Decomposing Agent Skill Quality into Interpretable Signals [27.757569384137767]
  General properties of the textttSKILL.md document play an important role in skill quality.<n>SkillEval learns an interpretable direction for each quality property from controlled positive--negative skill pairs.<n>SkillEval scores closely reflect downstream task performance, providing an early indication of whether a skill is likely to help an agent complete a task.
  arXiv  Detail & Related papers  (2026-08-07T07:25:17Z)
• What Is a Skill Worth? Structure-Aware Shapley Valuation of Agent Skills [18.13304162006297]
  We introduce SkillSV, a structure-aware framework for skill valuation.<n> SkillSV compiles a skill into units, dependencies, and hierarchy, so that only valid counterfactual skills are evaluated.<n>It estimates the resulting values with a rollout-budgeted estimator for noisy agent evaluations.
  arXiv  Detail & Related papers  (2026-08-05T07:56:26Z)
• A Framework for Evaluating Agentic Skills at Scale [0.4509560087514856]
  Agent skills are structured, reusable knowledge artifacts that augment agent capabilities.<n>No reusable methodology exists for evaluating an individual skill.<n>We present an evaluation framework that lets a skill author construct realistic tasks.
  arXiv  Detail & Related papers  (2026-06-16T11:46:56Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• Co-Evolving Skill Generation and Policy Optimization [35.41582114275514]
  Existing methods typically use strong language models to analyze trajectories, generate skills, and update a retrievable skill bank during online training.<n>We propose an online reinforcement learning framework for pre-storage skill validation.
  arXiv  Detail & Related papers  (2026-06-07T17:55:55Z)
• SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization [99.18305770138677]
  Agent skills are reusable strategies that guide agent reasoning and action.<n>Current skill construction methods treat the problem as one-shot extraction.<n>We introduce SkillComposer, a framework that decomposes skill construction into three learnable operations.
  arXiv  Detail & Related papers  (2026-06-04T12:16:08Z)
• OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents [15.598856888948093]
  We present textscOpenSkillEval, an automatic evaluation framework for both skill-augmented agent systems and the skills themselves.<n>Instead of relying on static benchmarks, textscOpenSkillEval automatically constructs realistic task instances from evolving real-world artifacts.<n>Using more than 600 dynamically generated task instances and 30 open-source skills, we conduct a systematic evaluation of state-of-the-art models and agent frameworks.
  arXiv  Detail & Related papers  (2026-05-22T14:09:41Z)
• From Context to Skills: Can Language Models Learn from Context Skillfully? [56.5857437218136]
  Ctx2Skill is a self-evolving framework that autonomously discovers, refines, and selects context-specific skills.<n>It can be plugged into any language model to obtain better context learning capability.
  arXiv  Detail & Related papers  (2026-04-30T09:53:15Z)
• Skill Retrieval Augmentation for Agentic AI [28.71123853744993]
  In existing agent systems, the dominant strategy for incorporating skills is to enumerate available skills within the context window.<n>This paper formulates Skill Retrieval Augmentation (SRA), a new paradigm in which agents dynamically retrieve, incorporate, and apply relevant skills from large external skill corpora on demand.<n>SRA-Bench is the first benchmark for evaluation of the full SRA pipeline, covering skill retrieval, skill incorporation, and end-task execution.
  arXiv  Detail & Related papers  (2026-04-27T15:19:59Z)
• SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement [66.44008181092832]
  LLM-based agent systems increasingly rely on agent skills sourced from open registries to extend their capabilities.<n>We introduce SkillAttack, a framework that verifies skill vulnerability exploitability through adversarial prompting.
  arXiv  Detail & Related papers  (2026-04-05T06:25:11Z)
• SkillNet: Create, Evaluate, and Connect AI Skills [159.47504178122156]
  SkillNet is an open infrastructure designed to create, evaluate, and organize AI skills at scale.<n>Our infrastructure integrates a repository of over 200,000 skills, an interactive platform, and a versatile Python toolkit.
  arXiv  Detail & Related papers  (2026-02-26T14:24:02Z)
• SoK: Agentic Skills -- Beyond Tool Use in LLM Agents [6.356997609995175]
  Agentic systems increasingly rely on reusable procedural capabilities, textita.k.a., agentic skills, to execute long-horizon reliably.<n>This paper maps the skill layer across the full lifecycle (discovery, practice, distillation, storage, composition, evaluation, and update)<n>We analyze the security and governance implications of skill-based agents, covering supply-chain risks, prompt injection via skill payloads, and trust-tiered execution.
  arXiv  Detail & Related papers  (2026-02-24T13:11:38Z)
• Skill-Based Reinforcement Learning with Intrinsic Reward Matching [77.34726150561087]
  We present Intrinsic Reward Matching (IRM), which unifies task-agnostic skill pretraining and task-aware finetuning. IRM enables us to utilize pretrained skills far more effectively than previous skill selection methods.
  arXiv  Detail & Related papers  (2022-10-14T00:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.