PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation

• URL: http://arxiv.org/abs/2412.12447v2
• Date: Fri, 20 Dec 2024 03:12:28 GMT
• Title: PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation
• Authors: Jaeseok Yoo, Hojae Han, Youngwon Lee, Jaejin Kim, Seung-won Hwang,
• Abstract summary: Plan-as-query Example Retrieval for few-shot prompting in Code generation (PERC)<n>PERC consistently outperforms the state-of-the-art RAG methods in code generation.
• Score: 20.105490443135672
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Code generation with large language models has shown significant promise, especially when employing retrieval-augmented generation (RAG) with few-shot examples. However, selecting effective examples that enhance generation quality remains a challenging task, particularly when the target programming language (PL) is underrepresented. In this study, we present two key findings: (1) retrieving examples whose presented algorithmic plans can be referenced for generating the desired behavior significantly improves generation accuracy, and (2) converting code into pseudocode effectively captures such algorithmic plans, enhancing retrieval quality even when the source and the target PLs are different. Based on these findings, we propose Plan-as-query Example Retrieval for few-shot prompting in Code generation (PERC), a novel framework that utilizes algorithmic plans to identify and retrieve effective examples. We validate the effectiveness of PERC through extensive experiments on the CodeContests, HumanEval and MultiPL-E benchmarks: PERC consistently outperforms the state-of-the-art RAG methods in code generation, both when the source and target programming languages match or differ, highlighting its adaptability and robustness in diverse coding environments.

Related papers

• Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency [0.0]
  We study how different programming activities compositions and training paradigms influence code generation effectiveness.<n>Our findings provide valuable insights for organizations seeking robust AI-driven coding solutions.
  arXiv  Detail & Related papers  (2025-05-04T14:44:27Z)
• Improving Large Language Model Planning with Action Sequence Similarity [50.52049888490524]
  In this work, we explore how to improve the model planning capability through in-context learning (ICL)<n>We propose GRASE-DC: a two-stage pipeline that first re-samples high AS exemplars and then curates the selected exemplars.<n>Our experimental result confirms that GRASE-DC achieves significant performance improvement on various planning tasks.
  arXiv  Detail & Related papers  (2025-05-02T05:16:17Z)
• From Token to Line: Enhancing Code Generation with a Long-Term Perspective [46.98293675904081]
  Large language models (LLMs) have significantly promoted the development of code generation task. We propose the textbfLSR-MCTS algorithm, which leverages MCTS to determine the code line-by-line and select the optimal path.
  arXiv  Detail & Related papers  (2025-04-10T04:03:25Z)
• Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMs [54.309127753635366]
  We present the results of a replication study in which we investigate GPT-4 effectiveness in recommending and suggesting idiomatic actions. Our findings underscore the potential of LLMs to achieve tasks where, in the past, implementing recommenders based on complex code analyses was required.
  arXiv  Detail & Related papers  (2025-01-28T15:41:54Z)
• Chain-of-Retrieval Augmented Generation [72.06205327186069]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities [19.455889970335967]
  Code generation aims to automatically generate code snippets of specific programming language according to natural language descriptions. One main challenge of pre-trained models for code generation is the semantic gap between natural language requirements and source code. Retrieval-augmented framework can be leveraged to help understand the requirements and provide guidance for the generation process.
  arXiv  Detail & Related papers  (2025-01-23T15:17:51Z)
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval [103.116634967815]
  We introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters. Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework. Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark.
  arXiv  Detail & Related papers  (2024-11-19T16:54:45Z)
• The First Prompt Counts the Most! An Evaluation of Large Language Models on Iterative Example-Based Code Generation [33.77058239791512]
  This paper presents the first comprehensive study on example-based code generation using Large Language Models (LLMs)<n>We adopt an iterative evaluation framework and formalize the objective of example-based code generation as two sequential sub-objectives.<n>We assess six state-of-the-art LLMs using a new benchmark of 172 diverse target functionalities.
  arXiv  Detail & Related papers  (2024-11-11T08:05:37Z)
• CodeDPO: Aligning Code Models with Self Generated and Verified Source Code [52.70310361822519]
  We propose CodeDPO, a framework that integrates preference learning into code generation to improve two key code preference factors: code correctness and efficiency. CodeDPO employs a novel dataset construction method, utilizing a self-generation-and-validation mechanism that simultaneously generates and evaluates code and test cases.
  arXiv  Detail & Related papers  (2024-10-08T01:36:15Z)
• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• SEED: Customize Large Language Models with Sample-Efficient Adaptation for Code Generation [35.88318116340547]
  We propose a novel adaptation approach named SEED, which stands for Sample-Efficient adaptation with Error-Driven learning for code generation. We show that SEED achieves superior performance with few training samples, showing an average relative improvement of 54.7% in Pass@1 on multiple code generation benchmarks.
  arXiv  Detail & Related papers  (2024-02-29T16:09:02Z)
• A Thorough Examination of Decoding Methods in the Era of LLMs [72.65956436513241]
  Decoding methods play an indispensable role in converting language models from next-token predictors into practical task solvers. This paper provides a comprehensive and multifaceted analysis of various decoding methods within the context of large language models. Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization.
  arXiv  Detail & Related papers  (2024-02-10T11:14:53Z)
• Stochastic Code Generation [1.7205106391379026]
  Large language models pre-trained for code generation can generate high-quality short code but often struggle with generating coherent long code. This issue is also observed in language modeling for long text generation. In this study, we investigate whether this technique can be applied to code generation to improve coherence.
  arXiv  Detail & Related papers  (2023-04-14T00:01:05Z)
• Execution-based Code Generation using Deep Reinforcement Learning [8.085533911328577]
  PPOCoder is a new framework for code generation that combines pre-trained PL models with Proximal Policy Optimization. PPOCoder seamlessly integrates external code-specific knowledge into the model optimization process. It's important to note that PPOCoder is a task-agnostic and model-agnostic framework that can be used across different code generation tasks and PLs.
  arXiv  Detail & Related papers  (2023-01-31T18:02:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.