RAG-Enhanced Commit Message Generation
- URL: http://arxiv.org/abs/2406.05514v2
- Date: Fri, 14 Jun 2024 06:23:37 GMT
- Title: RAG-Enhanced Commit Message Generation
- Authors: Linghao Zhang, Hongyi Zhang, Chong Wang, Peng Liang,
- Abstract summary: Commit Message Generation has become a research hotspot in automated software engineering.
We propose REACT, a novel REtrieval-Augmented framework for CommiT message generation.
- Score: 8.858678357308726
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Commit message is one of the most important textual information in software development and maintenance. However, it is time-consuming and labor-intensive to write commit messages manually. Commit Message Generation (CMG) has become a research hotspot in automated software engineering. Researchers have proposed several methods for CMG and achieved great results. In recent years, CodeBERT, CodeT5, and other Pre-trained Language Models (PLMs) for code have been proposed. These models can be easily transferred to code-related downstream tasks including CMG with simple fine-tuning and can achieve impressive performance. Moreover, Large Language Models (LLMs) with code capabilities (e.g., ChatGPT, Llama 3, Gemma) can be directly applied to various tasks by designing instruct prompts without training. This brings new possibilities to the CMG task. In this work, we propose REACT, a novel REtrieval-Augmented framework for CommiT message generation, which effectively integrates advanced retrieval techniques with different PLMs and LLMs and can broadly enhance the performance of various models on the CMG task. Specifically, we design and build a hybrid retriever to retrieve the most relevant code diff and commit message pair from the code base as an "exemplar". Then, the retrieved pair is utilized to guide and enhance the generation of commit messages by PLMs and LLMs through fine-tuning and in-context learning. Our approach is evaluated on a widely-used dataset. The experimental results show that REACT significantly enhances the performance of various models on the CMG task, improving the BLEU score of CodeT5 by up to 55%, boosting Llama 3's BLEU score by 102%, and substantially surpassing all baselines, achieving a new SOTA. This demonstrates the effectiveness and broad applicability of our framework that can enhance CMG by a large margin.
Related papers
- Input Conditioned Graph Generation for Language Agents [31.2175071107555]
We develop learnable and dynamic language agents using an existing framework that abstracts language agents as graphs.
We learn to generate edges that represent the flow of communication based on the given input, thereby adjusting the internal communication of a language agent.
Our approach surpasses the previous static approach by nearly 6% accuracy on a combined dataset of MMLU and CMMLU, and by more than 10% when trained with a sparsity-inducing loss.
arXiv Detail & Related papers (2024-06-17T13:53:15Z) - Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond [24.151927600694066]
Commit Message Generation (CMG) approaches aim to automatically generate commit messages based on given code diffs.
This paper conducts the first comprehensive experiment to investigate how far we have been in applying Large Language Models (LLMs) to generate high-quality commit messages.
arXiv Detail & Related papers (2024-04-23T08:24:43Z) - Meta-Task Prompting Elicits Embeddings from Large Language Models [54.757445048329735]
We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation.
We generate high-quality sentence embeddings from Large Language Models without the need for model fine-tuning.
Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.
arXiv Detail & Related papers (2024-02-28T16:35:52Z) - KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning
over Knowledge Graph [134.8631016845467]
We propose an autonomous LLM-based agent framework, called KG-Agent.
In KG-Agent, we integrate the LLM, multifunctional toolbox, KG-based executor, and knowledge memory.
To guarantee the effectiveness, we leverage program language to formulate the multi-hop reasoning process over the KG.
arXiv Detail & Related papers (2024-02-17T02:07:49Z) - Executable Code Actions Elicit Better LLM Agents [76.95566120678787]
This work proposes to use Python code to consolidate Large Language Model (LLM) agents' actions into a unified action space (CodeAct)
integrated with a Python interpreter, CodeAct can execute code actions and dynamically revise prior actions or emit new actions upon new observations through multi-turn interactions.
The encouraging performance of CodeAct motivates us to build an open-source LLM agent that interacts with environments by executing interpretable code and collaborates with users using natural language.
arXiv Detail & Related papers (2024-02-01T21:38:58Z) - Using Large Language Models for Commit Message Generation: A Preliminary
Study [5.5784148764236114]
Large language models (LLMs) can be used to generate commit messages automatically and effectively.
In 78% of the 366 samples, the commit messages generated by LLMs were evaluated by humans as the best.
arXiv Detail & Related papers (2024-01-11T14:06:39Z) - Chain-of-Thought in Neural Code Generation: From and For Lightweight
Language Models [23.727775288971003]
Large Language Models (LLMs) have demonstrated remarkable potential in code generation.
In this study, we investigate lightweight Language Models (lLMs) which are defined to have fewer than 10 billion parameters.
Based on these findings, we design a novel approach COTTON which can leverage lLMs to automatically generate Chain of Thought (CoTs)
The results show that the CoTs generated by COTTON outperform the baselines in terms of automated and human evaluation metrics.
arXiv Detail & Related papers (2023-12-09T12:20:50Z) - LLM-Assisted Code Cleaning For Training Accurate Code Generators [53.087019724256606]
We investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system.
We build a novel data-cleaning pipeline that uses these principles to transform existing programs.
We evaluate our approach on two challenging algorithmic code generation benchmarks and find that fine-tuning CodeLLaMa-7B improves the performance by up to 30% compared to fine-tuning on the original dataset.
arXiv Detail & Related papers (2023-11-25T02:45:50Z) - MAgIC: Investigation of Large Language Model Powered Multi-Agent in
Cognition, Adaptability, Rationality and Collaboration [102.41118020705876]
Large Language Models (LLMs) have marked a significant advancement in the field of natural language processing.
As their applications extend into multi-agent environments, a need has arisen for a comprehensive evaluation framework.
This work introduces a novel benchmarking framework specifically tailored to assess LLMs within multi-agent settings.
arXiv Detail & Related papers (2023-11-14T21:46:27Z) - CodeT5+: Open Code Large Language Models for Code Understanding and
Generation [72.1638273937025]
Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence.
CodeT5+ is a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks.
We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning.
arXiv Detail & Related papers (2023-05-13T14:23:07Z) - Better Language Models of Code through Self-Improvement [18.75015225501755]
We propose a simple data augmentation framework for pre-trained language models for code (PLMCs)
Our framework utilizes knowledge gained during the pre-training and fine-tuning stage to generate pseudo data, which is then used as training data for the next step.
The results show that our framework significantly improves PLMCs' performance in code-related sequence generation tasks.
arXiv Detail & Related papers (2023-04-02T10:59:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.