SWE-Explore: Benchmarking How Coding Agents Explore Repositories

• URL: http://arxiv.org/abs/2606.07297v1
• Date: Fri, 05 Jun 2026 14:08:27 GMT
• Title: SWE-Explore: Benchmarking How Coding Agents Explore Repositories
• Abstract summary: Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents.<n>In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration.
• Score: 20.73123882302075
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents. Yet they usually treat coding tasks as a holistic, binary prediction problem (e.g., resolved or unresolved), neglecting fine-grained agent capabilities such as repository understanding, context retrieval, code localization, and bug diagnosis. In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration, a critical capability of coding agents. Given a repository and an issue, SWE-Explore asks an explorer to return a ranked list of relevant code regions under a fixed line budget. SWE-Explore covers 848 issues across 10 programming languages and 203 open-source repositories. For each instance, we derive line-level ground truth from independent agent trajectories that successfully solved the same issue, distilling the specific code regions their solution paths actually consulted. We evaluate exploration along coverage, ranking, and context-efficiency dimensions, showing that these metrics strongly track downstream repair behavior. Across a broad set of retrieval methods, general coding agents, and specialized localizers, we find that agentic explorers form a clear tier above classical retrieval. While file-level localization is already strong for modern methods, line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers.

Related papers

• Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generation [53.85136813868056]
  ACToR is an adaptive critical token-aware retrieval framework for repository-level code generation.<n>We show that ACToR consistently outperforms state-of-the-art methods on repository-level benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T17:59:39Z)
• DeepRepoQA: Code Repository Question Answering with Deep Agent Exploration [10.655923413185498]
  DeepRepoQA is a novelQA framework for repository-level code understanding.<n>A Monte-Carlo Tree Search (MCTS) mechanism is employed to empower agents to dynamically search, navigate, and inspect code.
  arXiv  Detail & Related papers  (2026-08-25T08:26:31Z)
• CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding [44.75033190929879]
  CORE-Bench is a benchmark for code retrieval in the era of agentic coding.<n>It evaluates ability at three levels: code understanding, issue-to-edit localization, and broader context retrieval.<n> Experiments with representative embedding models show a sharp drop from traditional code search to code retrieval in agentic coding settings.
  arXiv  Detail & Related papers  (2026-06-10T09:38:41Z)
• RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations [51.43574078961796]
  We introduce RepoMirage, a two-stage evaluation suite built on SWE-Bench Verified.<n>RepoMirage-Perturb applies three types of semantics-preserving repository-level perturbations, revealing a clear performance drop when correct solving requires broader context access.<n>We propose RepoAnchor, a structure-first prototype workflow that separates repository exploration from downstream problem solving.
  arXiv  Detail & Related papers  (2026-05-25T06:26:43Z)
• Declarative Data Services: Structured Agentic Discovery for Composing Data Systems [1.8167057409188236]
  Declarative Data Services (DDS) is an architecture for structured agentic discovery of data-system compositions from declarative user intent.<n>As a proof of life on a trading-backend workload, converges where unbounded discovery does not; runtime failures become skill patches.
  arXiv  Detail & Related papers  (2026-05-20T04:36:40Z)
• BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? [61.247730037229815]
  We introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope.<n>To investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities.<n>This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
  arXiv  Detail & Related papers  (2026-03-03T17:52:01Z)
• AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion [55.21541958868449]
  We propose AlignCoder, a repository-level code completion framework.<n>Our framework generates an enhanced query that bridges the semantic gap between the initial query and the target code.<n>We employ reinforcement learning to train an AlignRetriever that learns to leverage inference information in the enhanced query for more accurate retrieval.
  arXiv  Detail & Related papers  (2026-01-27T15:23:14Z)
• SpIDER: Spatially Informed Dense Embedding Retrieval for Software Issue Localization [6.098008057625392]
  Agentic approaches typically employ sparse retrieval methods like BM25 or dense embedding strategies to identify relevant units.<n>We propose SpIDER (Spatially Informed Dense Embedding Retrieval), an enhanced dense retrieval approach that incorporates LLM-based reasoning over auxiliary context.<n> Empirical results show that SpIDER consistently improves dense retrieval performance across several programming languages.
  arXiv  Detail & Related papers  (2025-12-18T01:32:25Z)
• Improving Code Localization with Repository Memory [33.423769985220005]
  We introduce tools that allow the agent to retrieve from a non-parametric memory encompassing recent historical commits and linked issues.<n>We demonstrate that augmenting such a memory can significantly improve LocAgent, a state-of-the-art localization framework.
  arXiv  Detail & Related papers  (2025-10-01T15:10:15Z)
• CodeRAG: Finding Relevant and Necessary Knowledge for Retrieval-Augmented Repository-Level Code Completion [11.329578913209623]
  Repository-level code completion automatically predicts the unfinished code based on the broader information from the repository.<n>CodeRAG is a framework tailored to identify relevant and necessary knowledge for retrieval-augmented repository-level code completion.
  arXiv  Detail & Related papers  (2025-09-19T15:57:40Z)
• SweRank: Software Issue Localization with Code Ranking [109.3289316191729]
  SweRank is an efficient retrieve-and-rerank framework for software issue localization.<n>We construct SweLoc, a large-scale dataset curated from public GitHub repositories.<n>We show that SweRank achieves state-of-the-art performance, outperforming both prior ranking models and costly agent-based systems.
  arXiv  Detail & Related papers  (2025-05-07T19:44:09Z)
• CoIR: A Comprehensive Benchmark for Code Information Retrieval Models [52.61625841028781]
  COIR (Code Information Retrieval Benchmark) is a robust and comprehensive benchmark designed to assess code retrieval capabilities.<n>COIR comprises ten meticulously curated code datasets, spanning eight distinctive retrieval tasks across seven diverse domains.<n>We evaluate nine widely used retrieval models using COIR, uncovering significant difficulties in performing code retrieval tasks even with state-of-the-art systems.
  arXiv  Detail & Related papers  (2024-07-03T07:58:20Z)
• Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration [64.19431011897515]
  This paper presents Alibaba LingmaAgent, a novel Automated Software Engineering method designed to comprehensively understand and utilize whole software repositories for issue resolution.<n>Our approach introduces a top-down method to condense critical repository information into a knowledge graph, reducing complexity, and employs a Monte Carlo tree search based strategy.<n>In production deployment and evaluation at Alibaba Cloud, LingmaAgent automatically resolved 16.9% of in-house issues faced by development engineers, and solved 43.3% of problems after manual intervention.
  arXiv  Detail & Related papers  (2024-06-03T15:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.