SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

• URL: http://arxiv.org/abs/2603.15401v1
• Date: Mon, 16 Mar 2026 15:16:31 GMT
• Title: SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
• Abstract summary: SWE-Skills-Bench is the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering.<n>It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria.<n>Our results show that skill injection benefits are far more limited than rapid adoption suggests.
• Score: 27.583012125198568
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear. We present SWE-Skills-Bench, the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering (SWE). It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria, yielding approximately 565 task instances across six SWE subdomains. We introduce a deterministic verification framework that maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill. Our results show that skill injection benefits are far more limited than rapid adoption suggests: 39 of 49 skills yield zero pass-rate improvement, and the average gain is only +1.2%. Token overhead varies from modest savings to a 451% increase while pass rates remain unchanged. Only seven specialized skills produce meaningful gains (up to +30%), while three degrade performance (up to -10%) due to version-mismatched guidance conflicting with project context. These findings suggest that agent skills are a narrow intervention whose utility depends strongly on domain fit, abstraction level, and contextual compatibility. SWE-Skills-Bench provides a testbed for evaluating the design, selection, and deployment of skills in software engineering agents. SWE-Skills-Bench is available at https://github.com/GeniusHTX/SWE-Skills-Bench.

Related papers

• Signal or Noise? A Benchmark Study of Agent Skills in Web Development [6.7220476997481375]
  We use WebDev-Skills-Bench to study 31 public WebDev Skills on 50 Web-Bench projects and 1,000 ordered tasks.<n>Across four models, target Skill injection reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and increases token cost by 72% to 394%.<n>Losses concentrate on easy early tasks, Skill rankings transfer weakly across models, and anti-pattern rules outperform example-heavy content within helpful Skills.
  arXiv  Detail & Related papers  (2026-08-24T10:13:07Z)
• TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents [41.82249291285459]
  TRACE iteratively improves a skill-based agent's behavioral knowledge without modifying model weights.<n>On GPT-5.5, TRACE improves consistency (Pass3) by 34.6 points, from 59.9% to 94.5%, while shrinking the gap between potential and reliable performance to just 4.0 points.<n>These results show that TRACE converts high model potential into stable, consistent performance gain.
  arXiv  Detail & Related papers  (2026-08-24T04:36:03Z)
• Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills [9.077159743623076]
  repository-native framework for evaluating skills and product capability packages as executable agent artifacts.<n>ACES runs paired live trials with and without a target skill, grades six default runtime metrics, and reports Skill Lift.<n>Open-source implementation of the methodology is available in NVIDIA SkillEvaluator.
  arXiv  Detail & Related papers  (2026-08-20T23:26:14Z)
• SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents [16.77727447235193]
  We present SkillCorpus, a framework that aggregates, curates, matches, and evaluates the open skill ecosystem at scale.<n>It filters 821,000 crawled skills through a multi-stage pipeline into 96,401 skills organised by a 16-class taxonomy and three quality facets (utility, robustness, safety)<n>We evaluate end-to-end across three benchmarks (SkillsBench, GDPVal, QwenClawBench), two harnesses, and two open backbones with a frontier robustness check.
  arXiv  Detail & Related papers  (2026-07-17T01:55:17Z)
• From Registry to Repository: How AI Agent Skills Are Written, Adapted, and Maintained [13.991357454554949]
  We present the first empirical study of AI agent skills as engineered artefacts that are authored, reused, customised and maintained.<n>Most reused skills remain near-verbatim, 53% are never modified after adoption, and subsequent local maintenance is overwhelmingly additive.
  arXiv  Detail & Related papers  (2026-07-01T13:14:45Z)
• Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry [49.83108591873481]
  We study SKILL.md - only attacks across three registry-facing stages of the Agent Skill lifecycle.<n>We show that SKILL.md is not passive documentation but operational text that shapes which third-party capabilities agents find, trust, and use.
  arXiv  Detail & Related papers  (2026-05-12T02:11:54Z)
• EvoSkill: Automated Skill Discovery for Multi-Agent Systems [6.319876096746374]
  We introduce textbfEvoSkill, a self-evolving framework that automatically discovers and refines agent skills.<n>EvoSkill analyzes execution failures, proposes new skills or edits to existing ones, and materializes them into structured, reusable skill folders.<n>We evaluate EvoSkill on two benchmarks: OfficeQA, a grounded reasoning benchmark over U.S. Treasury data, and SealQA, a noisy retrieval benchmark.
  arXiv  Detail & Related papers  (2026-03-03T09:07:22Z)
• Hybrid-Gym: Training Coding Agents to Generalize Across Tasks [59.95803522351185]
  In this paper, we describe some transferable skills that are shared across diverse tasks.<n>We propose a training environment, Hybrid-Gym, consisting of a set of scalable synthetic tasks.<n>Experiments show that agents trained on our synthetic tasks effectively generalize diverse real-world tasks.
  arXiv  Detail & Related papers  (2026-02-18T19:30:55Z)
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks [61.89812116484928]
  Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time.<n>We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers.<n>We show curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain.<n>Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming.
  arXiv  Detail & Related papers  (2026-02-13T07:06:06Z)
• SWE-World: Building Software Engineering Agents in Docker-Free Environments [91.17484806743641]
  SWE-World is a Docker-free framework that replaces physical execution environments with a learned surrogate for training and evaluating software engineering agents.<n>We show that SWE-World raises Qwen2.5-Coder-32B from 6.2% to 52.0% via Docker-free SFT, 55.0% with Docker-free RL, and 68.2% with further TTS.
  arXiv  Detail & Related papers  (2026-02-03T11:44:39Z)
• SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Training [78.37721886775215]
  We present SWE-Master, an open-source framework for building effective software engineering agents.<n>SWE-Master systematically explores the complete agent development pipeline, including teacher-trajectory synthesis and data curation.<n>We evaluate SWE-Master on SWE-bench Verified, a standard benchmark for realistic software engineering tasks.
  arXiv  Detail & Related papers  (2026-02-03T11:38:48Z)
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason [1.6249398255272318]
  We present empirical evidence that performance gains on SWE-Bench-Verified may be partially driven by memorization rather than genuine problem-solving.<n>We show that state-of-the-art models achieve up to 76% accuracy in identifying buggy file paths using only issue descriptions, without access to repository structure.<n>These findings raise concerns about the validity of existing results and underscore the need for more robust, contamination-resistant benchmarks.
  arXiv  Detail & Related papers  (2025-06-14T00:25:26Z)
• R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents [32.06393076572057]
  AgentGym is the largest procedurally-curated executable gym environment for training real-world SWE-agents.<n>It is powered by two main contributions: SYNGEN, a synthetic data curation recipe, and Hybrid Test-time Scaling.<n>Overall, our approach achieves 51% on the SWE-Bench Verified benchmark, reflecting a new state-of-the-art for open-weight SWE-agents.
  arXiv  Detail & Related papers  (2025-04-09T17:55:19Z)
• Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents [106.87436596397816]
  Large language model (LLM) agents have shown great potential in solving real-world software engineering (SWE) problems. We propose DEI (Diversity Empowered Intelligence), a framework that leverages their unique expertise. Experiments show that a DEI-guided committee of agents is able to surpass the best individual agent's performance by a large margin.
  arXiv  Detail & Related papers  (2024-08-13T17:50:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.