SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents

• URL: http://arxiv.org/abs/2605.18693v1
• Date: Mon, 18 May 2026 17:28:36 GMT
• Title: SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
• Abstract summary: We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified protocol.<n>A generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses.<n> Experiments across a range of skill-generation methods and backbones show substantial performance variation.
• Score: 15.669318649782285
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether they can generate correct, reusable, and executable skills from repositories and documents. Existing benchmarks primarily evaluate the efficacy of given skills or the ability of agents to solve downstream tasks from raw context, but they do not isolate skill generation itself as the object of study. We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified and controlled protocol. In SkillGenBench, a generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses and assessed with unified evaluation procedures. The benchmark covers two generation regimes: task-conditioned generation, where a task-specific skill is synthesized after the task is revealed, and task-agnostic generation, where a reusable skill library must be distilled before downstream tasks are known. It also spans two complementary procedural sources: repository-grounded instances, where procedures are distributed across code, configuration, and scripts, and document-grounded instances, where procedures and constraints must be distilled from long-form text. We provide standardized task specifications, pinned environments, and evaluation protocols centered on deterministic execution-based checks, supplemented by auxiliary signals for diagnosis. Experiments across a range of skill-generation methods and backbones show substantial performance variation, highlight the difficulty of reusable skill distillation, and reveal distinct failure modes in skill generation from software repositories versus long-form documents. SkillGenBench establishes a reproducible testbed for studying skill generation as an independent research problem in agent systems.

Related papers

• GeoSkill:Experience-Driven Hierarchical Skill Learning with Collaborative Revision forGeospatialAgents [19.003018505935845]
  GeoSkill is an experience-driven hierarchical skill learning framework for geospatial agents.<n>It transforms historical execution experience into reusable hierarchical skills.<n>It improves both end-to-end task accuracy and tool-execution reliability in geospatial tasks.
  arXiv  Detail & Related papers  (2026-09-12T02:50:52Z)
• SkillAligner: Treating Retrieved Skills as Adaptable Drafts at Execution Time [14.34844805305673]
  General-purpose skills promise reusable procedural knowledge for language agents.<n>But semantic relevance does not guarantee execution utility.<n>We propose SkillAligner, a training-free execution-time skill adaptation framework.
  arXiv  Detail & Related papers  (2026-08-07T07:05:53Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• Anything2Skill: Compiling External Knowledge into Reusable Skills for Agents [27.965275812765015]
  Anything2Skill is a taxonomy-guided framework that compiles heterogeneous external knowledge into reusable, retrievable, and executable skills for agents.<n>We show that Anything2Skill combined with RAG achieves 98.85% and 94.10% success rates, respectively.
  arXiv  Detail & Related papers  (2026-06-08T10:24:32Z)
• SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization [99.18305770138677]
  Agent skills are reusable strategies that guide agent reasoning and action.<n>Current skill construction methods treat the problem as one-shot extraction.<n>We introduce SkillComposer, a framework that decomposes skill construction into three learnable operations.
  arXiv  Detail & Related papers  (2026-06-04T12:16:08Z)
• VASO: Formally Verifiable Self-Evolving Skills for Physical AI Agents [57.240036084348354]
  We introduce VASO, a framework for verification-guided self-evolution of robot skill contracts.<n>VASO verifies logically inconsistent skill contracts, then verifies plans induced by the skill against global and local temporal specifications.<n>On Clearpath Jackal and PX4 quadcopter tasks, VASO reaches 97.2% formal-specification compliance using fewer than 100 optimization samples.
  arXiv  Detail & Related papers  (2026-06-03T20:02:35Z)
• COLLEAGUE.SKILL: Automated AI Skill Generation via Expert Knowledge Distillation [58.84646485020439]
  We present an automated trace-to-skill distillation system for generating person-grounded AI skills.<n> COLLEAGUE.SKILL produces a versioned skill package with two tracks: a capability track for practices, mental models, and decisions, and a bounded behavior track for communication style, interaction rules, and correction history.
  arXiv  Detail & Related papers  (2026-05-29T12:59:08Z)
• SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills [31.23929961213889]
  Large language model (LLM) agents accumulate rich episodic trajectories while solving real-world tasks.<n>We introduce SkillEvolBench, a diagnostic benchmark for evaluating this step from experience reuse to skill formation.<n>We find that current agents often adapt locally but rarely form robust reusable skills.
  arXiv  Detail & Related papers  (2026-05-22T18:23:31Z)
• SkillRAE: Agent Skill-Based Context Compilation for Retrieval-Augmented Execution [10.83969404435554]
  Large Language Model (LLM)-based agents increasingly rely on reusable skill libraries to solve artifact-rich tasks.<n>We propose SkillRAE, a two-stage RAE approach focusing on skill-based context compilation.<n> Experiments on two public benchmarks show that SkillRAE achieves a significant improvement over baselines for RAE.
  arXiv  Detail & Related papers  (2026-05-11T07:31:48Z)
• Skill-R1: Agent Skill Evolution via Reinforcement Learning [84.35984979949502]
  Skill-R1 is a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards.<n>It preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates.
  arXiv  Detail & Related papers  (2026-05-10T06:19:15Z)
• Evidence Over Plans: Online Trajectory Verification for Skill Distillation [59.17625804241461]
  Posterior Distillation Index (PDI) is a trajectory-level metric that quantifies how well a distilled skill is grounded in the task-environment evidence.<n> SPARK generates environment-verified trajectories used to compute PDI.<n>We show that SPARK-generated skills consistently surpass no-skill baselines and outperform human-written skills on student models.
  arXiv  Detail & Related papers  (2026-05-09T22:15:13Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• Can Coding Agents Reproduce Findings in Computational Materials Science? [49.254975563645786]
  We present AutoMat, a benchmark for evaluating large language models' ability to reproduce scientific claims.<n>By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow.<n>Results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 54.1%.
  arXiv  Detail & Related papers  (2026-05-01T17:42:12Z)
• SKILLFOUNDRY: Building Self-Evolving Agent Skill Libraries from Heterogeneous Scientific Resources [5.7282243774488295]
  We present SkillFoundry, a self-evolving framework that converts resources into validated agent skills.<n>We show that SkillFoundry produces a substantially novel and internally valid skill library.<n>We also show that SkillFoundry can design new task-specific skills on demand for concrete scientific objectives.
  arXiv  Detail & Related papers  (2026-04-05T05:02:18Z)
• Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution [5.10403054516716]
  Existing datasets evaluate performance on isolated, single pull request (PR) tasks in a stateless manner.<n>We introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS.
  arXiv  Detail & Related papers  (2026-04-03T13:44:40Z)
• Agent psychometrics: Task-level performance prediction in agentic coding benchmarks [24.348135523715815]
  We present a framework for predicting success or failure on individual tasks tailored to the agentic coding regime.<n>Our approach augments Item Response Theory (IRT) with rich features extracted from tasks, including issue statements, repository contexts, solutions, and test cases.
  arXiv  Detail & Related papers  (2026-04-01T07:59:59Z)
• SoK: Agentic Skills -- Beyond Tool Use in LLM Agents [6.356997609995175]
  Agentic systems increasingly rely on reusable procedural capabilities, textita.k.a., agentic skills, to execute long-horizon reliably.<n>This paper maps the skill layer across the full lifecycle (discovery, practice, distillation, storage, composition, evaluation, and update)<n>We analyze the security and governance implications of skill-based agents, covering supply-chain risks, prompt injection via skill payloads, and trust-tiered execution.
  arXiv  Detail & Related papers  (2026-02-24T13:11:38Z)
• Self-Challenging Language Model Agents [98.62637336505242]
  We propose the Self-Challenging framework for training an agent on high-quality tasks that are generated by itself.<n>The framework achieves over a two-fold improvement in Llama-3.1-8B-Instruct, despite using only self-generated training data.
  arXiv  Detail & Related papers  (2025-06-02T14:23:33Z)
• Training Language Models to Generate Quality Code with Program Analysis Feedback [66.0854002147103]
  Code generation with large language models (LLMs) is increasingly adopted in production but fails to ensure code quality.<n>We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code.
  arXiv  Detail & Related papers  (2025-05-28T17:57:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.