Repo2Skill-Evo: Repository Skills Go Stale in Silence

• URL: http://arxiv.org/abs/2608.21964v1
• Date: Sat, 22 Aug 2026 14:07:51 GMT
• Title: Repo2Skill-Evo: Repository Skills Go Stale in Silence
• Abstract summary: Large language model (LLM) agents increasingly operate over evolving software repositories.<n>Prior work shows that they can improve agent performance.<n>We study whether agents can keep this externalized knowledge current.
• Score: 35.22867798999821
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, which scripts to run, and which conventions the current release expects. Agent skills externalize this knowledge into reusable units, and prior work shows that they can improve agent performance. What remains unclear is whether that improvement is durable. The same version specificity that makes a skill useful also makes it fragile: after a release, it may become stale without raising any explicit signal, while continuing to provide obsolete guidance. Externalizing knowledge into a skill can therefore make its decay invisible. We study whether agents can keep this externalized knowledge current. Repo2Skill-Evo casts each release transition as a skill-maintenance task: given a V1 skill set and the official V1-to-V2 patch, an agent must update obsolete skill content while preserving guidance that remains valid. Across 57 real-world repositories and 105 selected release transitions, every evaluated transition invalidates part of the V1 skill set. Yet six frontier agents reach only 29.9%-69.7% avg@3 macro F1 under a patch-grounded removal metric that balances stale-content recall against over-editing precision. Across runs, two opposing errors dominate: incomplete coverage of affected files in the skill set leaves stale content untouched, while overbroad editing is associated with higher recall but lower precision. Repository skills go stale in silence, and even frontier agents cannot reliably maintain them.

Related papers

• Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance [10.648969524722448]
  Recent work seeks to automate skill curation, but it largely evaluates against automated baselines and treats human maintenance as an unmeasured bottleneck.<n>We mine the full commit histories of five public AI-skill repositories, a purposive sample of AI-tooling organizations, covering 873 commits, 143 skill files, and 254 substantive post-creation edits from October 2025 to June 2026.<n>First, every substantive edit is authored or merged through a named human account, while 62% carry an AI co-author trailer, with large repository-level variation.
  arXiv  Detail & Related papers  (2026-09-04T19:18:18Z)
• Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills [86.15447717549154]
  We present DisCo, a skill-powered research agent that creates skills and uses them during research.<n>We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work.
  arXiv  Detail & Related papers  (2026-09-02T15:49:41Z)
• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• Anything2Skill: Compiling External Knowledge into Reusable Skills for Agents [27.965275812765015]
  Anything2Skill is a taxonomy-guided framework that compiles heterogeneous external knowledge into reusable, retrievable, and executable skills for agents.<n>We show that Anything2Skill combined with RAG achieves 98.85% and 94.10% success rates, respectively.
  arXiv  Detail & Related papers  (2026-06-08T10:24:32Z)
• COLLEAGUE.SKILL: Automated AI Skill Generation via Expert Knowledge Distillation [58.84646485020439]
  We present an automated trace-to-skill distillation system for generating person-grounded AI skills.<n> COLLEAGUE.SKILL produces a versioned skill package with two tracks: a capability track for practices, mental models, and decisions, and a bounded behavior track for communication style, interaction rules, and correction history.
  arXiv  Detail & Related papers  (2026-05-29T12:59:08Z)
• SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution [29.961197505726528]
  We present SkillsVote, a lifecycle-governance framework for Agent Skills.<n> SkillsVote profiles a million-scale open-source corpus for environment requirements, quality, and verifiability.<n>It decomposes trajectories into skill-linked subtasks, attributes outcomes to skill use, agent exploration, environment, and result signals, and admits only successful reusable discoveries to evidence-gated updates.
  arXiv  Detail & Related papers  (2026-05-18T13:44:19Z)
• SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs [70.1970574147839]
  We propose SKILLGRAPH, a framework that represents reusable skills as nodes in a directed graph.<n>SKILLGRAPH retrieves not just individual skills, but an ordered skill subgraph that can guide multi-step decision making.<n> Experiments show that SKILLGRAPH achieves state-of-the-art performance against memory-augmented RL methods.
  arXiv  Detail & Related papers  (2026-05-12T12:21:49Z)
• SkillOS: Learning Skill Curation for Self-Evolving Agents [67.94374107466957]
  We propose SkillOS, an experience-driven RL training recipe for learning skill curation in self-evolving agents.<n>SkillOS pairs a frozen agent executor that retrieves and applies skills with a trainable skill curator that updates an external SkillRepo from accumulated experience.<n>We show that SkillOS consistently outperforms memory-free and strong memory-based baselines in both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2026-05-07T17:31:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.