CoRaCMG: Contextual Retrieval-Augmented Framework for Commit Message Generation

• URL: http://arxiv.org/abs/2509.18337v1
• Date: Mon, 22 Sep 2025 19:00:04 GMT
• Title: CoRaCMG: Contextual Retrieval-Augmented Framework for Commit Message Generation
• Authors: Bo Xiong, Linghao Zhang, Chong Wang, Peng Liang,
• Abstract summary: Commit Message Generation aims to automatically generate descriptive commit messages from code diffs to reduce developers' effort and improve message quality.<n>We propose CoRaCMG, a Contextual Retrieval-augmented framework for Commit Message Generation, structured in three phases.<n>CoRaCMG enables LLMs to learn project-specific terminologies and writing styles from the retrieved diff-message pairs.
• Score: 17.392708936075223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Commit messages play a key role in documenting the intent behind code changes. However, they are often low-quality, vague, or incomplete, limiting their usefulness. Commit Message Generation (CMG) aims to automatically generate descriptive commit messages from code diffs to reduce developers' effort and improve message quality. Although recent advances in LLMs have shown promise in automating CMG, their performance remains limited. This paper aims to enhance CMG performance by retrieving similar diff-message pairs to guide LLMs to generate commit messages that are more precise and informative. We proposed CoRaCMG, a Contextual Retrieval-augmented framework for Commit Message Generation, structured in three phases: (1) Retrieve: retrieving the similar diff-message pairs; (2) Augment: combining them with the query diff into a structured prompt; and (3) Generate: generating commit messages corresponding to the query diff via LLMs. CoRaCMG enables LLMs to learn project-specific terminologies and writing styles from the retrieved diff-message pairs, thereby producing high-quality commit messages. We evaluated our method on various LLMs, including closed-source GPT models and open-source DeepSeek models. Experimental results show that CoRaCMG significantly boosts LLM performance across four metrics (BLEU, Rouge-L, METEOR, and CIDEr). Specifically, DeepSeek-R1 achieves relative improvements of 76% in BLEU and 71% in CIDEr when augmented with a single retrieved example pair. After incorporating the single example pair, GPT-4o achieves the highest improvement rate, with BLEU increasing by 89%. Moreover, performance gains plateau after more than three examples are used, indicating diminishing returns. Further analysis shows that the improvements are attributed to the model's ability to capture the terminologies and writing styles of human-written commit messages from the retrieved example pairs.

Related papers

• DiffuRank: Effective Document Reranking with Diffusion Language Models [71.16830004674513]
  We propose DiffuRank, a reranking framework built upon diffusion language models (dLLMs)<n>dLLMs support more flexible decoding and generation processes that are not constrained to a left-to-right order.<n>We show dLLMs achieve performance comparable to, and in some cases exceeding, that of autoregressive LLMs with similar model sizes.
  arXiv  Detail & Related papers  (2026-02-13T02:18:14Z)
• Contextual Code Retrieval for Commit Message Generation: A Preliminary Study [18.46986692375691]
  A commit message describes the main code changes in a commit and plays a crucial role in software maintenance.<n>Existing commit message generation approaches typically frame it as a direct mapping which inputs a code diff and produces a brief descriptive sentence as output.<n>We argue that relying solely on the code diff is insufficient, as raw code diff fails to capture the full context needed for generating high-quality commit messages.
  arXiv  Detail & Related papers  (2025-07-23T16:54:57Z)
• An Empirical Study on Commit Message Generation using LLMs via In-Context Learning [26.39743339039473]
  Commit messages concisely describe code changes in natural language.<n>We propose to borrow the weapon of large language models (LLMs) and in-context learning (ICL) to generate commit messages.
  arXiv  Detail & Related papers  (2025-02-26T07:47:52Z)
• Idiosyncrasies in Large Language Models [54.26923012617675]
  We unveil and study idiosyncrasies in Large Language Models (LLMs)<n>We find that fine-tuning text embedding models on LLM-generated texts yields excellent classification accuracy.<n>We leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies.
  arXiv  Detail & Related papers  (2025-02-17T18:59:02Z)
• Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation [81.18701211912779]
  We introduce an Adaptive Multi-Aspect Retrieval-augmented over KGs (Amar) framework.<n>This method retrieves knowledge including entities, relations, and subgraphs, and converts each piece of retrieved text into prompt embeddings.<n>Our method has achieved state-of-the-art performance on two common datasets.
  arXiv  Detail & Related papers  (2024-12-24T16:38:04Z)
• SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization [70.11167263638562]
  Social relation reasoning aims to identify relation categories such as friends, spouses, and colleagues from images. We first present a simple yet well-crafted framework named name, which combines the perception capability of Vision Foundation Models (VFMs) and the reasoning capability of Large Language Models (LLMs) within a modular framework.
  arXiv  Detail & Related papers  (2024-10-28T18:10:26Z)
• RAG-Enhanced Commit Message Generation [8.858678357308726]
  Commit Message Generation has become a research hotspot. It is time-consuming to write commit messages manually. This paper proposes REACT, a REtrieval-Augmented framework for CommiT message generation.
  arXiv  Detail & Related papers  (2024-06-08T16:24:24Z)
• Commit Messages in the Age of Large Language Models [0.9217021281095906]
  We evaluate the performance of OpenAI's ChatGPT for generating commit messages based on code changes. We compare the results obtained with ChatGPT to previous automatic commit message generation methods that have been trained specifically on commit data.
  arXiv  Detail & Related papers  (2024-01-31T06:47:12Z)
• Using Large Language Models for Commit Message Generation: A Preliminary Study [5.5784148764236114]
  Large language models (LLMs) can be used to generate commit messages automatically and effectively. In 78% of the 366 samples, the commit messages generated by LLMs were evaluated by humans as the best.
  arXiv  Detail & Related papers  (2024-01-11T14:06:39Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• Guiding Large Language Models via Directional Stimulus Prompting [114.84930073977672]
  We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model to generate an auxiliary directional stimulus prompt for each input instance.
  arXiv  Detail & Related papers  (2023-02-22T17:44:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.