RAG-Enhanced Commit Message Generation

• URL: http://arxiv.org/abs/2406.05514v3
• Date: Thu, 03 Oct 2024 17:15:34 GMT
• Title: RAG-Enhanced Commit Message Generation
• Authors: Linghao Zhang, Hongyi Zhang, Chong Wang, Peng Liang,
• Abstract summary: Commit Message Generation has become a research hotspot. It is time-consuming to write commit messages manually. This paper proposes REACT, a REtrieval-Augmented framework for CommiT message generation.
• Score: 8.858678357308726
• License:
• Abstract: Commit message is one of the most important textual information in software development and maintenance. However, it is time-consuming to write commit messages manually. Commit Message Generation (CMG) has become a research hotspot. Recently, several pre-trained language models (PLMs) and large language models (LLMs) with code capabilities have been introduced, demonstrating impressive performance on code-related tasks. Meanwhile, prior studies have explored the utilization of retrieval techniques for CMG, but it is still unclear what effects would emerge from combining advanced retrieval techniques with various generation models. This paper proposed REACT, a REtrieval-Augmented framework for CommiT message generation. It integrates advanced retrieval techniques with different PLMs and LLMs, to enhance the performance of these models on the CMG task. Specifically, a hybrid retriever is designed and used to retrieve the most relevant code diff and commit message pair as an exemplar. Then, the retrieved pair is utilized to guide and enhance the CMG task by PLMs and LLMs through fine-tuning and in-context learning. The experimental results show that REACT significantly enhances these models' performance on the CMG task, improving the BLEU score of CodeT5 by up to 55%, boosting Llama 3's BLEU score by 102%, and substantially surpassing all baselines.

Related papers

• Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings [77.20838441870151]
  Commit message generation is a crucial task in software engineering that is challenging to evaluate correctly. We use an online metric - the number of edits users introduce before committing the generated messages to the VCS - to select metrics for offline experiments. Our results indicate that edit distance exhibits the highest correlation, whereas commonly used similarity metrics such as BLEU and METEOR demonstrate low correlation.
  arXiv  Detail & Related papers  (2024-10-15T20:32:07Z)
• Instructive Code Retriever: Learn from Large Language Model's Feedback for Code Intelligence Tasks [10.867880635762395]
  We introduce a novel approach named Instructive Code Retriever (ICR) ICR is designed to retrieve examples that enhance model inference across various code intelligence tasks and datasets. We evaluate our model's effectiveness on various tasks, i.e., code summarization, program synthesis, and bug fixing.
  arXiv  Detail & Related papers  (2024-10-15T05:44:00Z)
• What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices [91.71951459594074]
  Long language models (LLMs) with extended context windows have significantly improved tasks such as information extraction, question answering, and complex planning scenarios. Existing methods typically utilize the Self-Instruct framework to generate instruction tuning data for better long context capability improvement. We propose the Multi-agent Interactive Multi-hop Generation framework, incorporating a Quality Verification Agent, a Single-hop Question Generation Agent, a Multiple Question Sampling Strategy, and a Multi-hop Question Merger Agent. Our findings show that our synthetic high-quality long-context instruction data significantly enhances model performance, even surpassing models trained on larger amounts of human
  arXiv  Detail & Related papers  (2024-09-03T13:30:00Z)
• Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond [24.151927600694066]
  Commit Message Generation (CMG) approaches aim to automatically generate commit messages based on given code diffs. This paper conducts the first comprehensive experiment to investigate how far we have been in applying Large Language Models (LLMs) to generate high-quality commit messages.
  arXiv  Detail & Related papers  (2024-04-23T08:24:43Z)
• Meta-Task Prompting Elicits Embeddings from Large Language Models [54.757445048329735]
  We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation. We generate high-quality sentence embeddings from Large Language Models without the need for model fine-tuning. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.
  arXiv  Detail & Related papers  (2024-02-28T16:35:52Z)
• LLM-Assisted Code Cleaning For Training Accurate Code Generators [53.087019724256606]
  We investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system. We build a novel data-cleaning pipeline that uses these principles to transform existing programs. We evaluate our approach on two challenging algorithmic code generation benchmarks and find that fine-tuning CodeLLaMa-7B improves the performance by up to 30% compared to fine-tuning on the original dataset.
  arXiv  Detail & Related papers  (2023-11-25T02:45:50Z)
• Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agents [9.191944519634111]
  The Georgetown InfoSense group has done in regard to solving the challenges presented by TREC iKAT 2023. Our submitted runs outperform the median runs by a significant margin, exhibiting superior performance in nDCG across various cut numbers and in overall success rate. Our solution involves the use of Large Language Models (LLMs) for initial answers, answer grounding by BM25, passage quality filtering by logistic regression, and answer generation by LLMs again.
  arXiv  Detail & Related papers  (2023-11-16T02:37:58Z)
• Enhancing Text-based Knowledge Graph Completion with Zero-Shot Large Language Models: A Focus on Semantic Enhancement [8.472388165833292]
  We introduce a framework termed constrained prompts for KGC (CP-KGC) This framework designs prompts that adapt to different datasets to enhance semantic richness. This study extends the performance limits of existing models and promotes further integration of KGC with large language models.
  arXiv  Detail & Related papers  (2023-10-12T12:31:23Z)
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
  A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
  arXiv  Detail & Related papers  (2023-05-31T03:18:03Z)
• MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrievers [140.0479479231558]
  In this work, we aim to unify a variety of pre-training tasks into a multi-task pre-trained model, namely MASTER. MASTER utilizes a shared-encoder multi-decoder architecture that can construct a representation bottleneck to compress the abundant semantic information across tasks into dense vectors.
  arXiv  Detail & Related papers  (2022-12-15T13:57:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.