Towards A Generalist Code Embedding Model Based On Massive Data Synthesis

• URL: http://arxiv.org/abs/2505.12697v1
• Date: Mon, 19 May 2025 04:37:53 GMT
• Title: Towards A Generalist Code Embedding Model Based On Massive Data Synthesis
• Authors: Chaofan Li, Jianlyu Chen, Yingxia Shao, Defu Lian, Zheng Liu,
• Abstract summary: We introduce textbfCodeR (underlineCode underlineRetrieval), a state-of-the-art embedding model for general-purpose code retrieval.<n>The superior performance of CodeR is built upon CodeR-Pile, a large-scale synthetic dataset constructed under the DRU principle.
• Score: 35.04242699869519
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code embedding models attract increasing attention due to the widespread popularity of retrieval-augmented generation (RAG) in software development. These models are expected to capture the rich semantic relationships inherent to code, which differ significantly from those found in text. However, existing models remain severely limited due to the scarcity of high-quality training data. In this work, we introduce \textbf{CodeR} (\underline{Code} \underline{R}etrieval), a state-of-the-art embedding model for general-purpose code retrieval. The superior performance of CodeR is built upon CodeR-Pile, a large-scale synthetic dataset constructed under the DRU (Diversity, Reliability, Usability) principle via a novel data synthesis pipeline. To optimize training effectiveness, we propose Annealing, a curriculum learning strategy that enables effective knowledge transfer across heterogeneous sources of data. We evaluate CodeR based on 16 diverse code retrieval tasks, where it significantly outperforms existing baselines and exhibits strong out-of-domain generalization performance. We have publicly released our code and the well-trained model to facilitate further research in this critical area. https://github.com/FlagOpen/FlagEmbedding/tree/master/research/BGE_Coder.

Related papers

• Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs [1.8838588087156363]
  This paper investigates the fine-tuning of code-generating Large Language Models (LLMs)<n>We enhance the training data for the reward model with the help of symbolic execution techniques.<n>Our reward models, fine-tuned on this dataset, demonstrate significant improvements over the baseline, CodeRL.
  arXiv  Detail & Related papers  (2025-04-21T16:29:07Z)
• Code Summarization Beyond Function Level [0.213063058314067]
  This study investigated the effectiveness of code summarization models beyond the function level.<n>The fine-tuned state-of-the-art CodeT5+ base model excelled in code summarization.<n> Repository-level summarization exhibited promising potential but requires significant computational resources.
  arXiv  Detail & Related papers  (2025-02-23T20:31:21Z)
• CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking [45.18877655831977]
  We introduce CoRNStack, a large-scale, high-quality contrastive training dataset for code that spans multiple programming languages.<n>This dataset is curated using consistency filtering to eliminate noisy positives and is further enriched with mined hard negatives.<n>We demonstrate that contrastive training of embedding models using CoRNStack leads to state-of-the-art performance across a variety of code retrieval tasks.
  arXiv  Detail & Related papers  (2024-12-01T23:54:12Z)
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval [103.116634967815]
  We introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters. Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework. Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark.
  arXiv  Detail & Related papers  (2024-11-19T16:54:45Z)
• OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models [76.59316249991657]
  Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent systems.<n>While open-access code LLMs are increasingly approaching the performance levels of proprietary models, high-quality code LLMs remain limited.<n>We introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an "open cookbook" for the research community.
  arXiv  Detail & Related papers  (2024-11-07T17:47:25Z)
• Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learning [29.135207235743795]
  This paper introduces VeriSeek, an LLM enhanced by reinforcement learning to achieve high Verilog code generation performance.<n>Our reinforcement learning approach employs code structure information as feedback signals to refine the pre-trained model.<n>Experiments show that VeriSeek outperforms state-of-the-art methods across multiple benchmarks.
  arXiv  Detail & Related papers  (2024-07-21T11:25:21Z)
• LLM-Assisted Code Cleaning For Training Accurate Code Generators [53.087019724256606]
  We investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system. We build a novel data-cleaning pipeline that uses these principles to transform existing programs. We evaluate our approach on two challenging algorithmic code generation benchmarks and find that fine-tuning CodeLLaMa-7B improves the performance by up to 30% compared to fine-tuning on the original dataset.
  arXiv  Detail & Related papers  (2023-11-25T02:45:50Z)
• Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders [63.28408887247742]
  We study whether training procedures can be improved to yield better generalization capabilities in the resulting models. We recommend a simple recipe for training dense encoders: Train on MSMARCO with parameter-efficient methods, such as LoRA, and opt for using in-batch negatives unless given well-constructed hard negatives.
  arXiv  Detail & Related papers  (2023-11-16T10:42:58Z)
• Source Code Data Augmentation for Deep Learning: A Survey [32.035973285175075]
  We conduct a comprehensive survey of data augmentation for source code. We highlight the general strategies and techniques to optimize the DA quality. We outline the prevailing challenges and potential opportunities for future research.
  arXiv  Detail & Related papers  (2023-05-31T14:47:44Z)
• CodeExp: Explanatory Code Document Generation [94.43677536210465]
  Existing code-to-text generation models produce only high-level summaries of code. We conduct a human study to identify the criteria for high-quality explanatory docstring for code. We present a multi-stage fine-tuning strategy and baseline models for the task.
  arXiv  Detail & Related papers  (2022-11-25T18:05:44Z)
• Enhancing Semantic Code Search with Multimodal Contrastive Learning and Soft Data Augmentation [50.14232079160476]
  We propose a new approach with multimodal contrastive learning and soft data augmentation for code search. We conduct extensive experiments to evaluate the effectiveness of our approach on a large-scale dataset with six programming languages.
  arXiv  Detail & Related papers  (2022-04-07T08:49:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.