Signal or Noise? A Benchmark Study of Agent Skills in Web Development

• URL: http://arxiv.org/abs/2608.23067v1
• Date: Mon, 24 Aug 2026 10:13:07 GMT
• Title: Signal or Noise? A Benchmark Study of Agent Skills in Web Development
• Abstract summary: We use WebDev-Skills-Bench to study 31 public WebDev Skills on 50 Web-Bench projects and 1,000 ordered tasks.<n>Across four models, target Skill injection reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and increases token cost by 72% to 394%.<n>Losses concentrate on easy early tasks, Skill rankings transfer weakly across models, and anti-pattern rules outperform example-heavy content within helpful Skills.
• Score: 6.7220476997481375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent Skills are reusable procedural modules that are increasingly injected into coding-agent sessions to encode framework conventions, anti-patterns, and reusable tools. However, because each injected Skill expands the prompt of every query, an effective Skill benchmark must determine not only whether an agent can solve a task, but whether the Skill should have been injected at all. We introduce WebDev-Skills-Bench and use it for a controlled empirical study of 31 public WebDev Skills on 50 Web-Bench projects and 1,000 ordered tasks. The benchmark compares four matched conditions, including a length-matched irrelevant control and leave-one-out component ablations. To isolate Skill effects from prompt-length artifacts, we place only SKILL.md in the prompt while mounting auxiliary files into the agent workspace. Across four models, target Skill injection reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and increases token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. Length-matched controls reveal two failure modes: some models are length-distracted, where an equally long irrelevant Skill reproduces most of the loss, while others are content-misled, where prompt length is neutral but Skill content still lowers Pass@2 by 1.1% to 1.4%. Further analysis shows that losses concentrate on easy early tasks, Skill rankings transfer weakly across models, and anti-pattern rules outperform example-heavy content within helpful Skills. These findings recast a matched Skill as a hypothesis about a particular Skill-project-model triple rather than a portable asset, reframing injection as a per-deployment routing decision and making length-matched controls and per-model audits a minimum standard for Agent-Skill evaluation.

Related papers

• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents [9.71657681051469]
  This paper presents a comprehensive analysis of skill-induced agent failures by attributing task failures and cost regressions to specific loaded skills.<n>We instantiate this framework on SkillsBench and SWE-Skills-Bench, yielding 307 skill-induced failures, including 125 functional failures and 182 efficiency regressions.<n>We propose research topics and tooling improvements for safer and more cost-aware skill reuse.
  arXiv  Detail & Related papers  (2026-08-12T10:15:19Z)
• Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning [72.09826781730662]
  Long-horizon reasoning demands that the model switch between distinct skills inside a reasoning chain.<n>We introduce Skill Entropy, a measure of the difficulty of switching from one skill to another.<n>We propose Skill-Entropy RL, an RL framework where the model predicts not only the answer at each step but also the skill used to produce it.
  arXiv  Detail & Related papers  (2026-08-05T17:57:16Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• SkillSmith: Compiling Agent Skills into Boundary-Guided Runtime Interfaces [6.384302247811717]
  In existing frameworks, skills are typically injected into the agent reasoning loop as contextual guidance once matched to a runtime task.<n>We find that this execution paradigm introduces two major sources of redundancy: irrelevant context injection and repeated skill-specific reasoning and planning.<n>We propose SkillSmith, a boundary-first compiler-runtime framework that compiles skill packages offline into minimal executable interfaces.
  arXiv  Detail & Related papers  (2026-05-12T09:25:25Z)
• Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry [49.83108591873481]
  We study SKILL.md - only attacks across three registry-facing stages of the Agent Skill lifecycle.<n>We show that SKILL.md is not passive documentation but operational text that shapes which third-party capabilities agents find, trust, and use.
  arXiv  Detail & Related papers  (2026-05-12T02:11:54Z)
• SkillRouter: Skill Routing for LLM Agents at Scale [18.540967600258607]
  Given a user task, the system must identify relevant skills before downstream planning or execution.<n>Existing agent stacks often rely on progressive disclosure, exposing only skill names and descriptions while hiding the full implementation body.<n>We present Skill, a compact 1.2B full-text retrieve-and-rerank pipeline.
  arXiv  Detail & Related papers  (2026-03-23T18:23:59Z)
• SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? [27.583012125198568]
  SWE-Skills-Bench is the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering.<n>It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria.<n>Our results show that skill injection benefits are far more limited than rapid adoption suggests.
  arXiv  Detail & Related papers  (2026-03-16T15:16:31Z)
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks [61.89812116484928]
  Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time.<n>We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers.<n>We show curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain.<n>Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming.
  arXiv  Detail & Related papers  (2026-02-13T07:06:06Z)
• Camouflage is all you need: Evaluating and Enhancing Language Model Robustness Against Camouflage Adversarial Attacks [53.87300498478744]
  Adversarial attacks represent a substantial challenge in Natural Language Processing (NLP) This study undertakes a systematic exploration of this challenge in two distinct phases: vulnerability evaluation and resilience enhancement. Results suggest a trade-off between performance and robustness, with some models maintaining similar performance while gaining robustness.
  arXiv  Detail & Related papers  (2024-02-15T10:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.