COMCAT: Leveraging Human Judgment to Improve Automatic Documentation and Summarization

• URL: http://arxiv.org/abs/2407.13648v1
• Date: Thu, 18 Jul 2024 16:26:31 GMT
• Title: COMCAT: Leveraging Human Judgment to Improve Automatic Documentation and Summarization
• Authors: Skyler Grandel, Scott Thomas Andersen, Yu Huang, Kevin Leach,
• Abstract summary: COMCAT is an approach to automate comment generation by augmenting Large Language Models with expertise-guided context. We develop the COMCAT pipeline to comment C/C++ files by (1) automatically identifying suitable locations in which to place comments, (2) predicting the most helpful type of comment for each location, and (3) generating a comment based on the selected location and comment type. In a human subject evaluation, we demonstrate that COMCAT-generated comments significantly improve developer code comprehension across three indicative software engineering tasks by up to 12% for 87% of participants.
• Score: 4.1491806566512235
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software maintenance constitutes a substantial portion of the total lifetime costs of software, with a significant portion attributed to code comprehension. Software comprehension is eased by documentation such as comments that summarize and explain code. We present COMCAT, an approach to automate comment generation by augmenting Large Language Models (LLMs) with expertise-guided context to target the annotation of source code with comments that improve comprehension. Our approach enables the selection of the most relevant and informative comments for a given snippet or file containing source code. We develop the COMCAT pipeline to comment C/C++ files by (1) automatically identifying suitable locations in which to place comments, (2) predicting the most helpful type of comment for each location, and (3) generating a comment based on the selected location and comment type. In a human subject evaluation, we demonstrate that COMCAT-generated comments significantly improve developer code comprehension across three indicative software engineering tasks by up to 12% for 87% of participants. In addition, we demonstrate that COMCAT-generated comments are at least as accurate and readable as human-generated comments and are preferred over standard ChatGPT-generated comments for up to 92% of snippets of code. Furthermore, we develop and release a dataset containing source code snippets, human-written comments, and human-annotated comment categories. COMCAT leverages LLMs to offer a significant improvement in code comprehension across a variety of human software engineering tasks.

Related papers

• AUTOGENICS: Automated Generation of Context-Aware Inline Comments for Code Snippets on Programming Q&A Sites Using LLM [1.971759811837406]
  Inline comments in source code facilitate easy comprehension, reusability, and enhanced readability. Code snippets in answers on Q&A sites like Stack Overflow (SO) often lack comments because answerers volunteer their time and often skip comments or explanations due to time constraints. Given these challenges, we introduced AUTOGENICS, a tool designed to integrate with SO to generate effective inline comments for code snippets in SO answers exploiting large language models.
  arXiv  Detail & Related papers  (2024-08-27T21:21:13Z)
• When simplicity meets effectiveness: Detecting code comments coherence with word embeddings and LSTM [6.417777780911223]
  Code comments play a crucial role in software development, as they provide programmers with practical information. Developers tend to leave comments unchanged after updating the code, resulting in a discrepancy between the two artifacts. It is crucial to identify if, given a code snippet, its corresponding comment is coherent and reflects well the intent behind the code.
  arXiv  Detail & Related papers  (2024-05-25T15:21:27Z)
• Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective [85.48043537327258]
  We propose MANGO (comMents As Natural loGic pivOts), including a comment contrastive training strategy and a corresponding logical comment decoding strategy. Results indicate that MANGO significantly improves the code pass rate based on the strong baselines. The robustness of the logical comment decoding strategy is notably higher than the Chain-of-thoughts prompting.
  arXiv  Detail & Related papers  (2024-04-11T08:30:46Z)
• Code Needs Comments: Enhancing Code LLMs with Comment Augmentation [91.52444946362547]
  We introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language. We conducted experiments on three code-focused Large Language Models and observed consistent improvements in performance on two widely-used programming skill benchmarks.
  arXiv  Detail & Related papers  (2024-02-20T13:56:38Z)
• Leveraging Generative AI: Improving Software Metadata Classification with Generated Code-Comment Pairs [0.0]
  In software development, code comments play a crucial role in enhancing code comprehension and collaboration. This research paper addresses the challenge of objectively classifying code comments as "Useful" or "Not Useful" We propose a novel solution that harnesses contextualized embeddings, particularly BERT, to automate this classification process.
  arXiv  Detail & Related papers  (2023-10-14T12:09:43Z)
• Exploring the Advances in Identifying Useful Code Review Comments [0.0]
  This paper reflects the evolution of research on the usefulness of code review comments. It examines papers that define the usefulness of code review comments, mine and annotate datasets, study developers' perceptions, analyze factors from different aspects, and use machine learning classifiers to automatically predict the usefulness of code review comments.
  arXiv  Detail & Related papers  (2023-07-03T00:41:20Z)
• Data-Driven Approach for Log Instruction Quality Assessment [59.04636530383049]
  There are no widely adopted guidelines on how to write log instructions with good quality properties. We identify two quality properties: 1) correct log level assignment assessing the correctness of the log level, and 2) sufficient linguistic structure assessing the minimal richness of the static text necessary for verbose event description. Our approach correctly assesses log level assignments with an accuracy of 0.88, and the sufficient linguistic structure with an F1 score of 0.99, outperforming the baselines.
  arXiv  Detail & Related papers  (2022-04-06T07:02:23Z)
• ReACC: A Retrieval-Augmented Code Completion Framework [53.49707123661763]
  We propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval. We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark.
  arXiv  Detail & Related papers  (2022-03-15T08:25:08Z)
• CoSQA: 20,000+ Web Queries for Code Search and Question Answering [63.92224685262063]
  CoSQA dataset includes 20,604 labels for pairs of natural language queries and codes. We introduce a contrastive learning method dubbed CoCLR to enhance query-code matching. We show that evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1%.
  arXiv  Detail & Related papers  (2021-05-27T15:37:21Z)
• Deep Just-In-Time Inconsistency Detection Between Comments and Source Code [51.00904399653609]
  In this paper, we aim to detect whether a comment becomes inconsistent as a result of changes to the corresponding body of code. We develop a deep-learning approach that learns to correlate a comment with code changes. We show the usefulness of our approach by combining it with a comment update model to build a more comprehensive automatic comment maintenance system.
  arXiv  Detail & Related papers  (2020-10-04T16:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.