CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding

• URL: http://arxiv.org/abs/2606.11864v1
• Date: Wed, 10 Jun 2026 09:38:41 GMT
• Title: CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding
• Abstract summary: CORE-Bench is a benchmark for code retrieval in the era of agentic coding.<n>It evaluates ability at three levels: code understanding, issue-to-edit localization, and broader context retrieval.<n> Experiments with representative embedding models show a sharp drop from traditional code search to code retrieval in agentic coding settings.
• Score: 44.75033190929879
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Code retrieval is becoming central to coding agents, but agentic coding requires more than matching a natural-language query to an isolated snippet. Given a user request, a coding agent needs to navigate a concrete repository state, locate relevant files and functions, gather supporting context, and filter similar in-repository distractors. Existing code retrieval benchmarks mainly evaluate docstring-to-function or snippet-level matching, thereby missing this requirement-driven repository search problem. To address this gap, we introduce CORE-Bench, a comprehensive benchmark for code retrieval in the era of agentic coding. CORE-Bench evaluates code retrieval ability at three levels: code understanding, issue-to-edit localization, and broader context retrieval. Built from curated code-search tasks and SWE-bench-series instances, CORE-Bench contains over 180K queries and 106K broader-context relevance labels. Experiments with representative embedding models show a sharp drop from traditional code search to code retrieval in agentic coding settings. Simple supervised fine-tuning of existing embedding models significantly improves performance in this setting, suggesting substantial room for further progress.

Related papers

• CodeHID: Learning an Addressable Hierarchical Code Index for Generative Code Retrieval [16.418964135916433]
  We propose CodeHID, a generative code retrieval framework that reformulates the code retrieval task from flat candidate matching to coarse-to-fine semantic address generation.<n>CodeHID relies on two core components. First, Pseudo-Neighbor Guided DocID Learning constructs a globally static hierarchical index by applying multi-level residual quantization and $k$-nearest-neighbor pseudo-supervision.<n>Second, Dual-Phase DocID Generation Guidance reliably navigates this fixed index by combining training-side ranking enhancements, using hard negatives and rank distillation, with inference-side candidate constraints and prefix-aware decoding.
  arXiv  Detail & Related papers  (2026-08-25T05:35:17Z)
• SWE-Explore: Benchmarking How Coding Agents Explore Repositories [20.73123882302075]
  Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents.<n>In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration.
  arXiv  Detail & Related papers  (2026-06-05T14:08:27Z)
• CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents [43.426809750160665]
  We show that a coding agent equipped with nothing more than a standard Unix terminal can be trained to achieve strong results.<n>Our work particularly focuses on techniques for re-purposing existing coding agent environments for code search, reward design, and RL optimization.<n>We release the resulting model family, CodeScout, along with all our code and data for the community to build upon.
  arXiv  Detail & Related papers  (2026-03-18T15:25:42Z)
• AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion [55.21541958868449]
  We propose AlignCoder, a repository-level code completion framework.<n>Our framework generates an enhanced query that bridges the semantic gap between the initial query and the target code.<n>We employ reinforcement learning to train an AlignRetriever that learns to leverage inference information in the enhanced query for more accurate retrieval.
  arXiv  Detail & Related papers  (2026-01-27T15:23:14Z)
• Beyond Function-Level Search: Repository-Aware Dual-Encoder Code Retrieval with Adversarial Verification [11.965887077524577]
  We introduce RepoAlign-Bench, the first benchmark designed to evaluate repository-level code retrieval under change request driven scenarios.<n>We propose ReflectCode, an adversarial reflection augmented dual-tower architecture featuring disentangled code_encoder and doc_encoder components.<n>Experiments demonstrate that ReflectCode achieves 12.2% improvement in Top-5 Accuracy and 7.1% in Recall over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-10-16T18:47:04Z)
• RANGER -- Repository-Level Agent for Graph-Enhanced Retrieval [0.38298911754746934]
  General-purpose automated software engineering (ASE) includes tasks such as code completion, retrieval, repair, QA, and summarization.<n>These tasks require a code retrieval system that can handle specific queries about code entities, or code entity queries.<n>We present RANGER, a repository-level code retrieval agent designed to address both query types.
  arXiv  Detail & Related papers  (2025-09-27T18:57:02Z)
• CodeRAG: Finding Relevant and Necessary Knowledge for Retrieval-Augmented Repository-Level Code Completion [11.329578913209623]
  Repository-level code completion automatically predicts the unfinished code based on the broader information from the repository.<n>CodeRAG is a framework tailored to identify relevant and necessary knowledge for retrieval-augmented repository-level code completion.
  arXiv  Detail & Related papers  (2025-09-19T15:57:40Z)
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval [103.116634967815]
  We introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters.<n>Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework.<n>Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark.
  arXiv  Detail & Related papers  (2024-11-19T16:54:45Z)
• CoIR: A Comprehensive Benchmark for Code Information Retrieval Models [52.61625841028781]
  COIR (Code Information Retrieval Benchmark) is a robust and comprehensive benchmark designed to assess code retrieval capabilities.<n>COIR comprises ten meticulously curated code datasets, spanning eight distinctive retrieval tasks across seven diverse domains.<n>We evaluate nine widely used retrieval models using COIR, uncovering significant difficulties in performing code retrieval tasks even with state-of-the-art systems.
  arXiv  Detail & Related papers  (2024-07-03T07:58:20Z)
• On The Importance of Reasoning for Context Retrieval in Repository-Level Code Editing [82.96523584351314]
  We decouple the task of context retrieval from the other components of the repository-level code editing pipelines. We conclude that while the reasoning helps to improve the precision of the gathered context, it still lacks the ability to identify its sufficiency.
  arXiv  Detail & Related papers  (2024-06-06T19:44:17Z)
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation [96.75695811963242]
  RepoCoder is a framework to streamline the repository-level code completion process. It incorporates a similarity-based retriever and a pre-trained code language model. It consistently outperforms the vanilla retrieval-augmented code completion approach.
  arXiv  Detail & Related papers  (2023-03-22T13:54:46Z)
• Generation-Augmented Query Expansion For Code Retrieval [51.20943646688115]
  We propose a generation-augmented query expansion framework. Inspired by the human retrieval process - sketching an answer before searching. We achieve new state-of-the-art results on the CodeSearchNet benchmark.
  arXiv  Detail & Related papers  (2022-12-20T23:49:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.