Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents

• URL: http://arxiv.org/abs/2609.12742v1
• Date: Fri, 11 Sep 2026 11:41:29 GMT
• Title: Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents
• Abstract summary: Coding agents increasingly read repository knowledge from SKILLs.<n>Recent work synthesizes these files automatically, by optimizing the document against a benchmark.<n>We mine harder tasks --- merged pull requests of the repository, reverted at a single frozen base commit; and score a candidate document by whether the same agent does better with it than without it.
• Score: 0.09558392439655013
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Coding agents increasingly read repository knowledge from SKILLs --- plain \texttt{.md} files versioned alongside the code. Recent work synthesizes these files automatically, by optimizing the document against a benchmark. A bare repository comes with no benchmark, and the synthetic tasks prior work builds are small enough that a capable agent saturates them with no document at all. We mine harder tasks --- merged pull requests of the repository, reverted at a single frozen base commit; and score a candidate document by whether the same agent does better with it than without it. On three Kotlin repositories, the documents GEPA finds raise this score by $4.9$pp on average, and the ones SkillOpt finds leave it where it started, $0.1$pp above the seed. The GEPA gain matches what prior work reports with the same optimizer, and at the dataset size a single repository supplies it cannot be separated from the agent's run-to-run variance; settling that would take more tasks than one repository's history yields. The documents themselves read better than the score: a maintainer of one repository found in them knowledge one only gets by working in the project.

Related papers

• Filesystem-Based Memory for LLM Agents: Organization, Evolution, and Sustainability [49.415740364852105]
  Deployed agents increasingly keep their long-term memory as a directory tree of markdown files that the agent itself reads, writes, and reorganizes through generic file tools.<n>Prior systems design bespoke memory representations and study retrieval over them, leaving the default's two working assumptions untested.<n>We present the first systematic exploration of chunk-based memory for LLM agents.
  arXiv  Detail & Related papers  (2026-07-29T08:59:43Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning [52.23991730630292]
  Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge.<n>We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer.<n>We introduce Agora, a benchmark pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M tokens, far exceeding any model's context window.
  arXiv  Detail & Related papers  (2026-06-23T12:57:18Z)
• Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA [7.478327602641997]
  Code-QA-Bench is a framework for synthesizing repository-level code understanding benchmarks.<n>We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories.<n>The framework is open-source and applicable to any well-documented Python repository.
  arXiv  Detail & Related papers  (2026-05-28T02:52:58Z)
• Remember Your Trace: Memory-Guided Long-Horizon Agentic Framework for Consistent and Hierarchical Repository-Level Code Documentation [27.40384564559557]
  We propose MemDocAgent, a framework that generates documentation within a single, integrated context spanning the entire repository.<n>MemDocAgent achieves the best performance over both open and closed-source baselines.
  arXiv  Detail & Related papers  (2026-05-14T08:35:20Z)
• BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases [0.0]
  We introduce BUILD-AND-FIND, a protocol for evaluating whether downstream agents can recover intended choices from generated repositories.<n>For each task, a builder sees a hidden repository specification and creates a specification-traced multiple-choice question bank.<n>The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort.
  arXiv  Detail & Related papers  (2026-05-07T12:35:27Z)
• RepoDoc: A Knowledge Graph-Based Framework to Automatic Documentation Generation and Incremental Updates [41.56228301287882]
  We propose RepoDoc, a system that uses a repository knowledge graph (RepoKG) as the semantic foundation for the entire documentation lifecycle.<n>For incremental updates, it cuts update time by 73% and token usage by 77%, and achieves 10.2% higher update recall.<n> Evaluated on 24 repositories across 8 languages, RepoDoc substantially outperforms state-of-the-art alternatives.
  arXiv  Detail & Related papers  (2026-04-29T10:43:02Z)
• MutaGReP: Execution-Free Repository-Grounded Plan Search for Code-Use [92.28400093066212]
  MutaGReP is an approach to search for plans that decompose a user request into natural language steps grounded in a large code repository.<n>Our plans use less than 5% of the 128K context window for GPT-4o but rival the coding performance of GPT-4o with a context window filled with the repo.
  arXiv  Detail & Related papers  (2025-02-21T18:58:17Z)
• Repository-level Code Search with Neural Retrieval Methods [25.222964965449286]
  We define the task of repository-level code search as retrieving the set of files from the current state of a code repository that are most relevant to addressing a user's question or bug.<n>The proposed approach combines BM25-based retrieval over commit messages with neural reranking using CodeBERT to identify the most pertinent files.<n> Experiments on a new dataset created from 7 popular open-source repositories demonstrate substantial improvements of up to 80% in MAP, MRR and P@1 over the BM25 baseline.
  arXiv  Detail & Related papers  (2025-02-10T21:59:01Z)
• Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration [64.19431011897515]
  This paper presents Alibaba LingmaAgent, a novel Automated Software Engineering method designed to comprehensively understand and utilize whole software repositories for issue resolution.<n>Our approach introduces a top-down method to condense critical repository information into a knowledge graph, reducing complexity, and employs a Monte Carlo tree search based strategy.<n>In production deployment and evaluation at Alibaba Cloud, LingmaAgent automatically resolved 16.9% of in-house issues faced by development engineers, and solved 43.3% of problems after manual intervention.
  arXiv  Detail & Related papers  (2024-06-03T15:20:06Z)
• RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation [79.83270415843857]
  We introduce RepoAgent, a large language model powered open-source framework aimed at proactively generating, maintaining, and updating code documentation. We have validated the effectiveness of our approach, showing that RepoAgent excels in generating high-quality repository-level documentation.
  arXiv  Detail & Related papers  (2024-02-26T15:39:52Z)
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation [96.75695811963242]
  RepoCoder is a framework to streamline the repository-level code completion process. It incorporates a similarity-based retriever and a pre-trained code language model. It consistently outperforms the vanilla retrieval-augmented code completion approach.
  arXiv  Detail & Related papers  (2023-03-22T13:54:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.