A Framework for Evaluating Agentic Skills at Scale

• URL: http://arxiv.org/abs/2606.17819v1
• Date: Tue, 16 Jun 2026 11:46:56 GMT
• Title: A Framework for Evaluating Agentic Skills at Scale
• Abstract summary: Agent skills are structured, reusable knowledge artifacts that augment agent capabilities.<n>No reusable methodology exists for evaluating an individual skill.<n>We present an evaluation framework that lets a skill author construct realistic tasks.
• Score: 0.4509560087514856
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent skills -- structured, reusable knowledge artifacts that augment LLM agent capabilities -- have been rapidly adopted in industry, yet their cross-domain impact and use across commercial and open-source models remain under-studied, and no reusable methodology exists for evaluating an individual skill. In this work, we present an evaluation framework that lets a skill author construct realistic tasks to rigorously assess the aspects of a skill that matter most to them, and that estimates skill utility by solving those tasks. Further, we apply our evaluation approach at scale to 500 real-world skills, generating 1,000 tasks derived from the skills' content, along with instruction-following and goal-completion scoring rubrics. Using these metrics, we evaluate how 19 agent-model configurations, both proprietary and open-source, perform on the tasks. Our results show that models vary widely in how closely they adhere to the instructions encoded in skills, leading to substantial differences in their performance gains. Furthermore, we show that access to a skill significantly changes model behavior compared to the no-skill setup, providing an essential mechanism for encoding opinionated workflows into LLM agents. We release our evaluation dataset to support future work on agent skills.

Related papers

• SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment [10.850946672552277]
  We introduce SkillAudit, an end-to-end framework for skill-centered assessment.<n>It generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety.<n>After scanning top-ranked real-world skill packages, we found that over 7% of skills are at risky status.
  arXiv  Detail & Related papers  (2026-06-21T17:40:05Z)
• SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization [99.18305770138677]
  Agent skills are reusable strategies that guide agent reasoning and action.<n>Current skill construction methods treat the problem as one-shot extraction.<n>We introduce SkillComposer, a framework that decomposes skill construction into three learnable operations.
  arXiv  Detail & Related papers  (2026-06-04T12:16:08Z)
• OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents [15.598856888948093]
  We present textscOpenSkillEval, an automatic evaluation framework for both skill-augmented agent systems and the skills themselves.<n>Instead of relying on static benchmarks, textscOpenSkillEval automatically constructs realistic task instances from evolving real-world artifacts.<n>Using more than 600 dynamically generated task instances and 30 open-source skills, we conduct a systematic evaluation of state-of-the-art models and agent frameworks.
  arXiv  Detail & Related papers  (2026-05-22T14:09:41Z)
• How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings [29.3744517703302]
  We study the first comprehensive study of skill utility under progressively challenging realistic settings.<n>Our findings reveal that performance gains degrade consistently as settings become more realistic.<n>We show that query-specific refinement substantially recovers lost performance when the initial skills are of reasonable relevance and quality.
  arXiv  Detail & Related papers  (2026-04-06T00:10:30Z)
• SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization [61.37478254738943]
  Inference-time skill augmentation is fundamentally limited.<n>We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization.<n>SKILL0 achieves substantial improvements over the standard RL baseline.
  arXiv  Detail & Related papers  (2026-04-02T17:03:05Z)
• SkillCraft: Can LLM Agents Learn to Use Tools Skillfully? [67.69996753743129]
  We introduce SkillCraft, a benchmark explicitly stress-test agent ability to form and reuse higher-level tool compositions.<n> SkillCraft features realistic, highly compositional tool-use scenarios with difficulty scaled along both quantitative and structural dimensions.<n>We propose a lightweight evaluation protocol that enables agents to auto-compose atomic tools into executable Skills, cache and reuse them inside and across tasks.
  arXiv  Detail & Related papers  (2026-02-28T15:44:31Z)
• SkillNet: Create, Evaluate, and Connect AI Skills [159.47504178122156]
  SkillNet is an open infrastructure designed to create, evaluate, and organize AI skills at scale.<n>Our infrastructure integrates a repository of over 200,000 skills, an interactive platform, and a versatile Python toolkit.
  arXiv  Detail & Related papers  (2026-02-26T14:24:02Z)
• Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward [5.124116559484265]
  The transition from monolithic language models to modular, skill-equipped agents marks a defining shift in how large language models (LLMs) are deployed in practice.<n>Rather than encoding all procedural knowledge within model weights, agent skills enable dynamic capability extension without retraining.<n>This survey provides a comprehensive treatment of the agent skills landscape, as it has rapidly evolved during the last few months.
  arXiv  Detail & Related papers  (2026-02-12T21:33:25Z)
• CATArena: Evaluation of LLM Agents through Iterative Tournament Competitions [49.02422075498554]
  Large Language Model (LLM) agents have evolved from basic text generation to autonomously completing complex tasks through interaction with external tools.<n>In this work, we emphasize the importance of learning ability, including both self-improvement and peer-learning, as a core driver for agent evolution toward human-level intelligence.<n>We propose an iterative, competitive peer-learning framework, which allows agents to refine and optimize their strategies through repeated interactions and feedback.
  arXiv  Detail & Related papers  (2025-10-30T15:22:53Z)
• Latent-Predictive Empowerment: Measuring Empowerment without a Simulator [56.53777237504011]
  We present Latent-Predictive Empowerment (LPE), an algorithm that can compute empowerment in a more practical manner. LPE learns large skillsets by maximizing an objective that is a principled replacement for the mutual information between skills and states.
  arXiv  Detail & Related papers  (2024-10-15T00:41:18Z)
• Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives [54.14429346914995]
  Chain-of-Thought (CoT) has become a pivotal method for solving complex problems with large language models (LLMs)<n>This paper introduces the Re-TASK framework, a novel theoretical model that revisits LLM tasks from capability, skill, and knowledge perspectives.<n> Experiments across diverse domains demonstrate the effectiveness of Re-TASK.
  arXiv  Detail & Related papers  (2024-08-13T13:58:23Z)
• Design of Negative Sampling Strategies for Distantly Supervised Skill Extraction [19.43668931500507]
  We propose an end-to-end system for skill extraction, based on distant supervision through literal matching. We observe that using the ESCO taxonomy to select negative examples from related skills yields the biggest improvements. We release the benchmark dataset for research purposes to stimulate further research on the task.
  arXiv  Detail & Related papers  (2022-09-13T13:37:06Z)
• Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach [40.06500618820166]
  This paper presents an approach to deriving a learner model directly from an assessment rubric. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills.
  arXiv  Detail & Related papers  (2022-09-07T10:09:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.