Beyond Retrieval: A Multitask Benchmark and Model for Code Search

• URL: http://arxiv.org/abs/2605.04615v2
• Date: Fri, 08 May 2026 04:03:59 GMT
• Title: Beyond Retrieval: A Multitask Benchmark and Model for Code Search
• Abstract summary: textscCoREB is built from counterfactually rewritten LiveCodeBench problems in five programming languages.<n>We benchmark eleven embedding models and five rerankers across three tasks: text-to-code, code-to-text, and code-to-code.<n>textscCoREB-Reranker is the first to achieve consistent gains across all three tasks.
• Score: 21.20978098597112
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code search has usually been evaluated as first-stage retrieval, even though production systems rely on broader pipelines with reranking and developer-style queries. Existing benchmarks also suffer from data contamination, label noise, and degenerate binary relevance. In this paper, we introduce \textsc{CoREB}, a contamination-limited, multitask \underline{co}de \underline{r}etrieval and r\underline{e}ranking \underline{b}enchmark, together with a fine-tuned code reranker, that goes beyond retrieval to cover the full code search pipeline. \textsc{CoREB} is built from counterfactually rewritten LiveCodeBench problems in five programming languages and delivered as timed releases with graded relevance judgments. We benchmark eleven embedding models and five rerankers across three tasks: text-to-code, code-to-text, and code-to-code. Our experiments reveal that: \circone code-specialised embeddings dominate code-to-code retrieval (${\sim}2{\times}$ over general encoders), yet no single model wins all three tasks; \circtwo short keyword queries, the format closest to real developer search, collapse every model to near-zero nDCG@10; \circthree off-the-shelf rerankers are task-asymmetric, with a 12-point swing on code-to-code and no baseline net-positive across all tasks; \circfour our fine-tuned \textsc{CoREB-Reranker} is the first to achieve consistent gains across all three tasks. The data and model are released.

Related papers

• TraceCoder: Explainable and Auditable Code Generation with Position-Key Snippet Versioning [0.0]
  We present a code generation concept that addresses shortcomings through three complementary mechanisms.<n>A relational snippet-history schema records, per repair event, the benchmark reference, round number, failure text, and explanation.<n>A browser-based visualisation tool renders this history as heat-mapped, hover-annotated source code.
  arXiv  Detail & Related papers  (2026-07-28T22:03:52Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding [44.75033190929879]
  CORE-Bench is a benchmark for code retrieval in the era of agentic coding.<n>It evaluates ability at three levels: code understanding, issue-to-edit localization, and broader context retrieval.<n> Experiments with representative embedding models show a sharp drop from traditional code search to code retrieval in agentic coding settings.
  arXiv  Detail & Related papers  (2026-06-10T09:38:41Z)
• Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves [1.0312968200748116]
  Large language models can generate substantial code and draft research text, but research-software projects require more than either artifact alone.<n>We identify two LM-specific failure modes: hallucination accumulation and desynchronization.<n>We propose Comet-H, an iterative prompt that orchestrates ideation, implementation, evaluation, grounding, and paper-writing as coordinates of a single workspace state.
  arXiv  Detail & Related papers  (2026-04-29T21:28:17Z)
• RANGER -- Repository-Level Agent for Graph-Enhanced Retrieval [0.38298911754746934]
  General-purpose automated software engineering (ASE) includes tasks such as code completion, retrieval, repair, QA, and summarization.<n>These tasks require a code retrieval system that can handle specific queries about code entities, or code entity queries.<n>We present RANGER, a repository-level code retrieval agent designed to address both query types.
  arXiv  Detail & Related papers  (2025-09-27T18:57:02Z)
• MGS3: A Multi-Granularity Self-Supervised Code Search Framework [22.214324677526132]
  We introduce a novel Multi-Granularity Self-Supervised contrastive learning code Search framework (MGS$3$)<n>First, MGS$3$ features a Supervised Multi-Granularity Representation module (HMGR), which aggregates fine-grained information into coarser-grained representations.<n>We conduct extensive experiments on code search benchmarks across various granularities, demonstrating that the framework exhibits outstanding performance in code search tasks of multiple granularities.
  arXiv  Detail & Related papers  (2025-05-30T06:49:39Z)
• KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding [49.56049319037421]
  KodCode is a synthetic dataset that addresses the persistent challenge of acquiring high-quality, verifiable training data.<n>It comprises question-solution-test triplets that are systematically validated via a self-verification procedure.<n>This pipeline yields a large-scale, robust and diverse coding dataset.
  arXiv  Detail & Related papers  (2025-03-04T19:17:36Z)
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval [103.116634967815]
  We introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters.<n>Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework.<n>Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark.
  arXiv  Detail & Related papers  (2024-11-19T16:54:45Z)
• Let the Code LLM Edit Itself When You Edit the Code [50.46536185784169]
  underlinetextbfPositional textbfIntegrity textbfEncoding (PIE)<n>PIE reduces computational overhead by over 85% compared to the standard full recomputation approach.<n>Results demonstrate that PIE reduces computational overhead by over 85% compared to the standard full recomputation approach.
  arXiv  Detail & Related papers  (2024-07-03T14:34:03Z)
• CoIR: A Comprehensive Benchmark for Code Information Retrieval Models [52.61625841028781]
  COIR (Code Information Retrieval Benchmark) is a robust and comprehensive benchmark designed to assess code retrieval capabilities.<n>COIR comprises ten meticulously curated code datasets, spanning eight distinctive retrieval tasks across seven diverse domains.<n>We evaluate nine widely used retrieval models using COIR, uncovering significant difficulties in performing code retrieval tasks even with state-of-the-art systems.
  arXiv  Detail & Related papers  (2024-07-03T07:58:20Z)
• R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models [62.20537000942005]
  We propose the R2C2-Coder to enhance and benchmark the real-world repository-level code completion abilities of code Large Language Models.<n>R2C2-Coder includes a code prompt construction method R2C2-Enhance and a well-designed benchmark R2C2-Bench.
  arXiv  Detail & Related papers  (2024-06-03T14:24:29Z)
• Improving Code Search with Hard Negative Sampling Based on Fine-tuning [15.341959871682981]
  We introduce a cross-encoder architecture for code search that jointly encodes the concatenation of query and code. We also introduce a Retriever-Ranker (RR) framework that cascades the dual-encoder and cross-encoder to promote the efficiency of evaluation and online serving.
  arXiv  Detail & Related papers  (2023-05-08T07:04:28Z)
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation [96.75695811963242]
  RepoCoder is a framework to streamline the repository-level code completion process. It incorporates a similarity-based retriever and a pre-trained code language model. It consistently outperforms the vanilla retrieval-augmented code completion approach.
  arXiv  Detail & Related papers  (2023-03-22T13:54:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.