SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents

• URL: http://arxiv.org/abs/2607.15557v4
• Date: Thu, 23 Jul 2026 10:15:08 GMT
• Title: SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents
• Abstract summary: We present SkillCorpus, a framework that aggregates, curates, matches, and evaluates the open skill ecosystem at scale.<n>It filters 821,000 crawled skills through a multi-stage pipeline into 96,401 skills organised by a 16-class taxonomy and three quality facets (utility, robustness, safety)<n>We evaluate end-to-end across three benchmarks (SkillsBench, GDPVal, QwenClawBench), two harnesses, and two open backbones with a frontier robustness check.
• Score: 16.77727447235193
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agent skills, SKILL files that package reusable procedural knowledge for an LLM agent, are a popular mechanism for extending agent capabilities. Public repositories now host them in large and growing numbers, yet these artifacts are fragmented, redundant, and uneven in quality, and their value in practice is unclear. A core question remains open, namely how to consolidate this open-source SKILL ecosystem into a single usable corpus, and what bounds its benefit on real-world agent tasks. We present SkillCorpus, a framework that aggregates, curates, matches, and evaluates the open skill ecosystem at scale. It filters ~821,000 crawled skills through a multi-stage pipeline into 96,401 skills organised by a 16-class taxonomy and three quality facets (utility, robustness, safety), and pairs them with a fine-tuned retrieval-and-selection stack that matches task-relevant skills. We evaluate end-to-end across three benchmarks (SkillsBench, GDPVal, QwenClawBench), two harnesses, and two open backbones with a frontier robustness check. Integrating SkillCorpus yields consistent gains across all three benchmarks, largest on SkillsBench (+7.5 pp). An operational analysis traces the gains to a coverage boundary and a harness boundary. SkillCorpus is, to our knowledge, the first end-to-end account of when a curated, retrieval-served community corpus improves real agent tasks, and where it does not. The dataset, models, and code will be released upon acceptance.

Related papers

• Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills [86.15447717549154]
  We present DisCo, a skill-powered research agent that creates skills and uses them during research.<n>We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work.
  arXiv  Detail & Related papers  (2026-09-02T15:49:41Z)
• Generative Skill Composition for LLM Agents [58.47191648555256]
  LLM agents benefit from skills for solving complex tasks.<n>As skill libraries grow, selecting an appropriate skill composition has emerged as a central bottleneck.<n>We propose SkillComposer, which instantiates structured skill composition as task-conditioned skill sequence prediction.
  arXiv  Detail & Related papers  (2026-06-30T17:53:09Z)
• SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment [10.850946672552277]
  We introduce SkillAudit, an end-to-end framework for skill-centered assessment.<n>It generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety.<n>After scanning top-ranked real-world skill packages, we found that over 7% of skills are at risky status.
  arXiv  Detail & Related papers  (2026-06-21T17:40:05Z)
• VISUALSKILL: Multimodal Skills for Computer-Use Agents [63.79539541125141]
  We propose VISUALSKILL: a hierarchical multimodal skill tailored to each target application.<n>We construct each skill with a two-stage pipeline that combines authored documentation with live-application UI exploration.<n>On two CUA benchmarks, a Claude Code CLI agent backed by Claude Opus 4.6 reaches an average score of 0.456 with VISUALSKILL.
  arXiv  Detail & Related papers  (2026-06-16T19:57:07Z)
• SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization [99.18305770138677]
  Agent skills are reusable strategies that guide agent reasoning and action.<n>Current skill construction methods treat the problem as one-shot extraction.<n>We introduce SkillComposer, a framework that decomposes skill construction into three learnable operations.
  arXiv  Detail & Related papers  (2026-06-04T12:16:08Z)
• AcademiClaw: When Students Set Challenges for AI Agents [69.70303995121154]
  AcademiClaw is a benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic disconnects.<n>We show that even the best of models achieves only a 55% pass rate.<n>We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community.
  arXiv  Detail & Related papers  (2026-05-04T14:40:42Z)
• SkillX: Automatically Constructing Skill Knowledge Bases for Agents [27.56549031749266]
  SkillX is a fully automated framework for constructing a plug-and-play skill knowledge base.<n>It operates through a fully automated pipeline built on three synergistic innovations.<n> Experiments show that SkillKB consistently improves task success and execution efficiency when plugged into weaker base agents.
  arXiv  Detail & Related papers  (2026-04-06T16:09:33Z)
• SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization [61.37478254738943]
  Inference-time skill augmentation is fundamentally limited.<n>We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization.<n>SKILL0 achieves substantial improvements over the standard RL baseline.
  arXiv  Detail & Related papers  (2026-04-02T17:03:05Z)
• SkillRouter: Skill Routing for LLM Agents at Scale [18.540967600258607]
  Given a user task, the system must identify relevant skills before downstream planning or execution.<n>Existing agent stacks often rely on progressive disclosure, exposing only skill names and descriptions while hiding the full implementation body.<n>We present Skill, a compact 1.2B full-text retrieve-and-rerank pipeline.
  arXiv  Detail & Related papers  (2026-03-23T18:23:59Z)
• SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? [27.583012125198568]
  SWE-Skills-Bench is the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering.<n>It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria.<n>Our results show that skill injection benefits are far more limited than rapid adoption suggests.
  arXiv  Detail & Related papers  (2026-03-16T15:16:31Z)
• Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale [28.43462779191672]
  AgentSkillOS is a principled framework for skill selection, orchestration, and ecosystem-level management.<n>AgentSkillOS comprises two stages: (i) Manage Skills, which organizes skills into a capability tree.<n> (ii) Solve Tasks, which retrieves, orchestrates, and executes multiple skills through DAG-based pipelines.
  arXiv  Detail & Related papers  (2026-03-02T18:46:47Z)
• SkillNet: Create, Evaluate, and Connect AI Skills [159.47504178122156]
  SkillNet is an open infrastructure designed to create, evaluate, and organize AI skills at scale.<n>Our infrastructure integrates a repository of over 200,000 skills, an interactive platform, and a versatile Python toolkit.
  arXiv  Detail & Related papers  (2026-02-26T14:24:02Z)
• SoK: Agentic Skills -- Beyond Tool Use in LLM Agents [6.356997609995175]
  Agentic systems increasingly rely on reusable procedural capabilities, textita.k.a., agentic skills, to execute long-horizon reliably.<n>This paper maps the skill layer across the full lifecycle (discovery, practice, distillation, storage, composition, evaluation, and update)<n>We analyze the security and governance implications of skill-based agents, covering supply-chain risks, prompt injection via skill payloads, and trust-tiered execution.
  arXiv  Detail & Related papers  (2026-02-24T13:11:38Z)
• Cross-modal Retrieval Models for Stripped Binary Analysis [62.89251403093734]
  BinSeek is the first two-stage cross-modal retrieval framework for stripped binary code analysis.<n>It consists of two models: BinSeekEmbedding is trained on large-scale dataset to learn the semantic relevance of the binary code.<n>BinSeek-Reranker learns to carefully judge the relevance of the candidate code to the description with context augmentation.
  arXiv  Detail & Related papers  (2025-12-11T07:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.