The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents

• URL: http://arxiv.org/abs/2607.22520v1
• Date: Fri, 24 Jul 2026 17:50:03 GMT
• Title: The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents
• Abstract summary: Adding procedural skills to an LLM agent is typically evaluated by average improvement in task success.<n>This metric hides an important cost: skills can also make agents worse.<n>We measure both sides by comparing agents with and without skills across nearly 6,000 runs.
• Score: 0.12891210250935145
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adding procedural skills to an LLM agent is typically evaluated by average improvement in task success. However, this metric hides an important cost: skills can also make agents worse. We measure both sides by comparing agents with and without skills across nearly 6,000 runs spanning two office automation benchmarks and three model harness stacks. This allows us to distinguish two outcomes. A regression is a task solved without skills but failed after skills are added. A residual failure is a task that fails both with and without skills. We find that regressions are substantial enough that the best performing skills outperform others primarily by regressing less, not by gaining more. We identify three causes of regression: (i) skill description osmosis, a skill changes an agent's behavior simply by being present in context, even when it is never invoked; (ii) grounding displacement, a skill's prescribed procedure overrides how the agent interprets its inputs; and (iii) verification displacement, where the procedure suppresses checks the agent would otherwise perform on its outputs. Analysing persistent failures reveals the same underlying pattern. Existing skills overemphasize procedural guidance the stage least often responsible for failure while under supporting grounding and verification, the dominant sources of remaining errors. After correcting evaluation artifacts and studying traces, we find many regressions and persistent failures recoverable through better grounding and verification. Procedural skills should be evaluated by decomposing their net effect into gains and regressions, not by aggregate improvement alone. We identify three regression modes skills should avoid, and find that reliability depends more on grounding and verification than on procedural skill choice.

Related papers

• When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents [62.10623192614306]
  Extracting skills from past successes is critical for the efficient evolution of Large Language Model (LLM) agents.<n>We propose textbfBoundary-Aware Skill Memory (BASM), which augments each skill with explicit boundary fields.<n>BASM consistently outperforms success-distilled skill-memory baselines.
  arXiv  Detail & Related papers  (2026-08-23T10:16:06Z)
• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents [9.71657681051469]
  This paper presents a comprehensive analysis of skill-induced agent failures by attributing task failures and cost regressions to specific loaded skills.<n>We instantiate this framework on SkillsBench and SWE-Skills-Bench, yielding 307 skill-induced failures, including 125 functional failures and 182 efficiency regressions.<n>We propose research topics and tooling improvements for safer and more cost-aware skill reuse.
  arXiv  Detail & Related papers  (2026-08-12T10:15:19Z)
• FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents [49.519184792774304]
  FailForge is an agentic framework that converts failed rollouts into training signal.<n>FailForge recovers over 26% of previously failed instances at marginal additional cost.
  arXiv  Detail & Related papers  (2026-08-09T08:22:57Z)
• AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows [1.6785141970297952]
  AEVAL (Agentic Evaluation) is a CI-integrated framework that replaces this practice with a deterministic, reproducible test pipeline for agentic skills.<n>Every skill change triggers a test event: the skill runs against a developer declared evaluation contract inside an automated executor.<n>A key ingredient is a structural separation between executor and grader, preventing a subtle but pervasive failure mode.
  arXiv  Detail & Related papers  (2026-07-16T21:33:05Z)
• When Good Verifiers Go Bad: Self-Improving VLMs Can Regress on New Tasks [2.743683637024251]
  Verifier-driven self-DPO is a common recipe for self-improving production visual-language models.<n>We show that this assumption can fail because verifier quality is highly task-specific.<n>We give a compact mechanistic explanation via a variance theorem for progress-gated replay and its direction-mismatch failure mode.
  arXiv  Detail & Related papers  (2026-06-12T16:55:30Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior [23.11308948097101]
  Current benchmarks rarely distinguish what a Skill says from how it is organized.<n>We study this distinction through Progressive Disclosure, where a concise root file points agents to supporting resources on demand.<n>We present SkillJuror, a framework for evaluating Skill writing paradigms.
  arXiv  Detail & Related papers  (2026-06-10T01:11:50Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs [59.62441340813425]
  We study whether reinforcement learning can improve direct recall of parametric knowledge.<n>Across three model families and multiple factual QA benchmarks, RL yields 27% average relative gains.<n>Mechanistically, RL primarily redistributes probability mass over existing knowledge rather than acquiring new facts.
  arXiv  Detail & Related papers  (2026-05-08T02:40:12Z)
• Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations [103.16279860448874]
  We propose an online reinforcement learning method using a novel binary retrieval-augmented reward (RAR)<n>For open-ended generation, binary RAR achieves a 39.3% reduction in hallucination rates.<n>In short-form question answering, the model learns abstention, strategically outputting "I don't know" when faced with insufficient parametric knowledge.
  arXiv  Detail & Related papers  (2025-10-20T16:45:43Z)
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty [59.97939500426759]
  This paper describes RLCR, an approach to training reasoning models that jointly improves accuracy and confidence estimation.<n>We show that across diverse datasets, RLCR substantially improves calibration with no loss in accuracy.<n>We also demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration.
  arXiv  Detail & Related papers  (2025-07-22T17:56:01Z)
• R-Tuning: Instructing Large Language Models to Say `I Don't Know' [66.11375475253007]
  Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. Previous instruction tuning methods force the model to complete a sentence no matter whether the model knows the knowledge or not. We present a new approach called Refusal-Aware Instruction Tuning (R-Tuning) Experimental results demonstrate R-Tuning effectively improves a model's ability to answer known questions and refrain from answering unknown questions.
  arXiv  Detail & Related papers  (2023-11-16T08:45:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.