EVOR: Evolving Retrieval for Code Generation

• URL: http://arxiv.org/abs/2402.12317v2
• Date: Tue, 03 Dec 2024 15:56:26 GMT
• Title: EVOR: Evolving Retrieval for Code Generation
• Authors: Hongjin Su, Shuyang Jiang, Yuhang Lai, Haoyuan Wu, Boao Shi, Che Liu, Qian Liu, Tao Yu,
• Abstract summary: Existing pipelines for retrieval-augmented code generation employ static knowledge bases with a single source.<n>We develop a novel pipeline, EVOR, that employs the synchronous evolution of both queries and diverse knowledge bases.
• Score: 17.46870626157077
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently the retrieval-augmented generation (RAG) has been successfully applied in code generation. However, existing pipelines for retrieval-augmented code generation (RACG) employ static knowledge bases with a single source, limiting the adaptation capabilities of Large Language Models (LLMs) to domains they have insufficient knowledge of. In this work, we develop a novel pipeline, EVOR, that employs the synchronous evolution of both queries and diverse knowledge bases. On two realistic settings where the external knowledge is required to solve code generation tasks, we compile four new datasets associated with frequently updated libraries and long-tail programming languages, named EVOR-BENCH. Extensive experiments demonstrate that EVOR achieves two to four times of execution accuracy compared to other methods such as Reflexion (Shinn et al., 2024), DocPrompting (Zhou et al., 2023), etc. We demonstrate that EVOR is flexible and can be easily combined with them to achieve further improvement. Further analysis reveals that EVOR benefits from the synchronous evolution of queries and documents and the diverse information sources in the knowledge base. We hope that our studies will inspire more insights into the design of advanced RACG pipelines in future research. Our model, code, and data are available at https://arks-codegen.github.io.

Related papers

• What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond [32.467437657603604]
  Repository-level code generation remains challenging due to complex code dependencies and the limitations of large language models (LLMs) in processing long contexts. We propose AllianceCoder, a novel context-integrated method that employs chain-of-thought prompting to decompose user queries into implementation steps and retrieves APIs via semantic description matching. Through extensive experiments on CoderEval and RepoExec, AllianceCoder achieves state-of-the-art performance, improving Pass@1 by up to 20% over existing approaches.
  arXiv  Detail & Related papers  (2025-03-26T14:41:38Z)
• CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale [39.54772602678732]
  This paper introduces CODESYNC, a data engine for identifying outdated code patterns. Building upon CODESYNC, we develop CODESYNCBENCH, a benchmark for assessing Large Language Models' ability to stay synchronized with code evolution.
  arXiv  Detail & Related papers  (2025-02-23T16:46:18Z)
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval [103.116634967815]
  We introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters. Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework. Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark.
  arXiv  Detail & Related papers  (2024-11-19T16:54:45Z)
• Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report [3.4632900249241874]
  This paper presents an experience report on the development of Retrieval Augmented Generation (RAG) systems using PDF documents as the primary data source. The RAG architecture combines generative capabilities of Large Language Models (LLMs) with the precision of information retrieval. The practical implications of this research lie in enhancing the reliability of generative AI systems in various sectors.
  arXiv  Detail & Related papers  (2024-10-21T12:21:49Z)
• CodeRAG-Bench: Can Retrieval Augment Code Generation? [78.37076502395699]
  We conduct a systematic, large-scale analysis of code generation using retrieval-augmented generation. We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks. We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources.
  arXiv  Detail & Related papers  (2024-06-20T16:59:52Z)
• VersiCode: Towards Version-controllable Code Generation [58.82709231906735]
  Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development. We propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM) We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge.
  arXiv  Detail & Related papers  (2024-06-11T16:15:06Z)
• SelfEvolve: A Code Evolution Framework via Large Language Models [5.6607714367826105]
  Large language models (LLMs) have already revolutionized code generation, after being pretrained on publicly available code data. We propose a novel two-step pipeline, called autoknow, that leverages LLMs as both knowledge providers and self-reflective programmers. We evaluate autoknowon three code generation datasets, including DS-1000 for data science code, HumanEval for software engineering code, and TransCoder for C++-to-Python translation.
  arXiv  Detail & Related papers  (2023-06-05T14:12:46Z)
• Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy [164.83371924650294]
  We show that strong performance can be achieved by a method we call Iter-RetGen, which synergizes retrieval and generation in an iterative manner. A model output shows what might be needed to finish a task, and thus provides an informative context for retrieving more relevant knowledge. Iter-RetGen processes all retrieved knowledge as a whole and largely preserves the flexibility in generation without structural constraints.
  arXiv  Detail & Related papers  (2023-05-24T16:17:36Z)
• Synergistic Interplay between Search and Large Language Models for Information Retrieval [141.18083677333848]
  InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections. InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-12T11:58:15Z)
• Active Retrieval Augmented Generation [123.68874416084499]
  Augmenting large language models (LMs) by retrieving information from external knowledge resources is one promising solution. Most existing retrieval augmented LMs employ a retrieve-and-generate setup that only retrieves information once based on the input. We propose Forward-Looking Active REtrieval augmented generation (FLARE), a generic method which iteratively uses a prediction of the upcoming sentence to anticipate future content.
  arXiv  Detail & Related papers  (2023-05-11T17:13:40Z)
• REINFOREST: Reinforcing Semantic Code Similarity for Cross-Lingual Code Search Models [11.78036105494679]
  This paper introduces a novel code-to-code search technique that enhances the performance of Large Language Models (LLMs) We present the first-ever code search method that encodes dynamic information during training without the need to execute either the corpus under search or the search query at inference time.
  arXiv  Detail & Related papers  (2023-05-05T20:46:56Z)
• Generate rather than Retrieve: Large Language Models are Strong Context Generators [74.87021992611672]
  We present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer.
  arXiv  Detail & Related papers  (2022-09-21T01:30:59Z)
• Enhancing Semantic Code Search with Multimodal Contrastive Learning and Soft Data Augmentation [50.14232079160476]
  We propose a new approach with multimodal contrastive learning and soft data augmentation for code search. We conduct extensive experiments to evaluate the effectiveness of our approach on a large-scale dataset with six programming languages.
  arXiv  Detail & Related papers  (2022-04-07T08:49:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.