Not All Skills Help: Measuring and Repairing Agent Knowledge

• URL: http://arxiv.org/abs/2606.15390v1
• Date: Sat, 13 Jun 2026 16:37:38 GMT
• Title: Not All Skills Help: Measuring and Repairing Agent Knowledge
• Abstract summary: ASSAY is a framework that separates generation from curation.<n>It computes a per-skill causal attribution on a small development set.<n>It restructures the library offline, and suppresses skills with negative predicted effect for each test task.
• Score: 47.498075849595374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents can improve without weight updates by accumulating natural-language skills from experience, but current systems entrust every decision about which skills to keep and how to apply them to LLM judgment alone. We argue that this conflates two distinct roles: generating a skill from experience is a creative act that judgment handles well, while deciding whether that skill actually helps requires empirical evidence across many tasks. Measuring per-skill causal contributions via randomized masking, we find that skill libraries exhibit pervasive causal heterogeneity: individual skills routinely help on some task types while hurting on others, yet their opposing effects cancel in aggregate, making them invisible to global curation methods. We propose ASSAY, a framework that separates generation from curation: it computes a per-skill causal attribution on a small development set, restructures the library offline, and suppresses skills with negative predicted effect for each test task. Across seven base models spanning four providers and two benchmarks (AppWorld and tau-bench), ASSAY consistently improves over prior skill-curation approaches. On AppWorld's hardest split, DeepSeek-V3 achieves 69.3% task-goal completion (47.4% relative improvement), a new state of the art among all published methods including weight-tuned approaches. On tau-bench retail, GPT-4.1 improves by 8.7% relative, advancing past o4-mini, o1, and GPT-4.5 on the public leaderboard without any weight modification. Ablation traces the dominant gain to per-task masking, confirming that the bottleneck is matching skills to tasks at inference time, not removing bad skills globally. Code is available at https://github.com/aiming-lab/assay.

Related papers

• When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents [62.10623192614306]
  Extracting skills from past successes is critical for the efficient evolution of Large Language Model (LLM) agents.<n>We propose textbfBoundary-Aware Skill Memory (BASM), which augments each skill with explicit boundary fields.<n>BASM consistently outperforms success-distilled skill-memory baselines.
  arXiv  Detail & Related papers  (2026-08-23T10:16:06Z)
• Write, Execute, Refine: From Skill Followers to Skill Optimizers via Reinforcement Learning from Execution Feedback [52.78692582507693]
  Expert-written natural language skills can improve tool-using agents, yet agent-authored skills perform 8-11 points worse than using no skill.<n>We study how to organize execution experience from intermediate skills into training states for an executor.
  arXiv  Detail & Related papers  (2026-08-18T09:52:48Z)
• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning [72.09826781730662]
  Long-horizon reasoning demands that the model switch between distinct skills inside a reasoning chain.<n>We introduce Skill Entropy, a measure of the difficulty of switching from one skill to another.<n>We propose Skill-Entropy RL, an RL framework where the model predicts not only the answer at each step but also the skill used to produce it.
  arXiv  Detail & Related papers  (2026-08-05T17:57:16Z)
• Progressive Agent Skill Generation via Reinforcement Learning [29.782307460602876]
  We propose Skill-$$, a reinforcement learning method for progressively generating high-quality agent skills.<n>Specifically, we formulate skill generation as a sequential editing process that decomposes skill construction into individually evaluable edits.<n>Experiments show that Skill-$$ generates more effective skills than methods based on pipeline-style consolidation.
  arXiv  Detail & Related papers  (2026-08-03T04:14:59Z)
• Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows [0.0]
  Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family.<n>Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck.<n>We ask whether LLM-generated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone?
  arXiv  Detail & Related papers  (2026-07-08T15:00:16Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• SkillCAT: Contrastive Assessment and Topology-Aware Skill Self-Evolution for LLM Agents [42.30624901249764]
  Current pipelines learn from one trajectory per task, merge candidate skill patches before checking them, and load the full skill corpus before inference.<n>We propose SkillCAT, a training-free framework that separates this process into three stages.<n>We evaluate SkillCAT on common agent benchmarks, including SpreadsheetBench, WikiTableQuestions, and DocVQA.
  arXiv  Detail & Related papers  (2026-06-11T13:12:10Z)
• SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization [99.18305770138677]
  Agent skills are reusable strategies that guide agent reasoning and action.<n>Current skill construction methods treat the problem as one-shot extraction.<n>We introduce SkillComposer, a framework that decomposes skill construction into three learnable operations.
  arXiv  Detail & Related papers  (2026-06-04T12:16:08Z)
• SkillsInjector: Dynamic Skill Context Construction for LLM Agents [16.631471381875816]
  Existing methods treat skill injection as a static step, selecting skills with fixed criteria, fixing the budget in advance, and leaving descriptions unchanged.<n>We propose SkillsInjector, a two-stage adaptive method that jointly addresses these decisions.<n>Across tau2-bench, SkillsBench, and ALFWorld, SkillsInjector achieves the highest score, improving over the strongest baseline by 3.9, 6.1, and 7.3 percentage points, respectively.
  arXiv  Detail & Related papers  (2026-05-28T11:44:32Z)
• MOCHA: Multi-Objective Chebyshev Annealing for Agent Skill Optimization [35.57702110117083]
  MOCHA (Multi-language Chebyshev Annealing) covers the full front, including non-resident regions.<n> MOCHA breaks through on every task, achieving 7.5% relative improvement in mean correctness over the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-19T04:07:41Z)
• SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs [70.1970574147839]
  We propose SKILLGRAPH, a framework that represents reusable skills as nodes in a directed graph.<n>SKILLGRAPH retrieves not just individual skills, but an ordered skill subgraph that can guide multi-step decision making.<n> Experiments show that SKILLGRAPH achieves state-of-the-art performance against memory-augmented RL methods.
  arXiv  Detail & Related papers  (2026-05-12T12:21:49Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• SkillLens: Adaptive Multi-Granularity Skill Reuse for Cost-Efficient LLM Agents [6.293920920196533]
  SkillLens is a hierarchical skill-evolution framework that organizes skills into a four-layer graph of policies, strategies, procedures, and primitives.<n>It retrieves semantically relevant skill seeds, expands them through degree-corrected random walk over the skill graph, and then uses a verifier to decide whether each visited unit should be accepted, decomposed, rewritten, or skipped.<n>Across MuLocbench and ALFWorld, SkillLens consistently improves over strong skill-based baselines.
  arXiv  Detail & Related papers  (2026-05-08T18:48:04Z)
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks [61.89812116484928]
  Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time.<n>We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers.<n>We show curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain.<n>Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming.
  arXiv  Detail & Related papers  (2026-02-13T07:06:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.