Explaining Explanation: An Empirical Study on Explanation in Code Reviews

• URL: http://arxiv.org/abs/2311.09020v2
• Date: Thu, 10 Oct 2024 16:58:53 GMT
• Title: Explaining Explanation: An Empirical Study on Explanation in Code Reviews
• Authors: Ratnadira Widyasari, Ting Zhang, Abir Bouraffa, Walid Maalej, David Lo,
• Abstract summary: We study the types of explanations used in code reviews and explore the potential of Large Language Models (LLMs) We extracted 793 code review comments from Gerrit and manually labeled them based on whether they contained a suggestion, an explanation, or both. Our analysis shows that 42% of comments only include suggestions without explanations.
• Score: 17.005837826213416
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code reviews are central for software quality assurance. Ideally, reviewers should explain their feedback to enable authors of code changes to understand the feedback and act accordingly. Different developers might need different explanations in different contexts. Therefore, assisting this process first requires understanding the types of explanations reviewers usually provide. The goal of this paper is to study the types of explanations used in code reviews and explore the potential of Large Language Models (LLMs), specifically ChatGPT, in generating these specific types. We extracted 793 code review comments from Gerrit and manually labeled them based on whether they contained a suggestion, an explanation, or both. Our analysis shows that 42% of comments only include suggestions without explanations. We categorized the explanations into seven distinct types including rule or principle, similar examples, and future implications. When measuring their prevalence, we observed that some explanations are used differently by novice and experienced reviewers. Our manual evaluation shows that, when the explanation type is specified, ChatGPT can correctly generate the explanation in 88 out of 90 cases. This foundational work highlights the potential for future automation in code reviews, which can assist developers in sharing and obtaining different types of explanations as needed, thereby reducing back-and-forth communication.

Related papers

• Leveraging Reward Models for Guiding Code Review Comment Generation [13.306560805316103]
  Code review is a crucial component of modern software development, involving the evaluation of code quality, providing feedback on potential issues, and refining the code to address identified problems.<n>Deep learning techniques are able to tackle the generative aspect of code review, by commenting on a given code as a human reviewer would do.<n>In this paper, we introduce CoRAL, a deep learning framework automating review comment generation by exploiting reinforcement learning with a reward mechanism.
  arXiv  Detail & Related papers  (2025-06-04T21:31:38Z)
• Code Review Comprehension: Reviewing Strategies Seen Through Code Comprehension Theories [12.81041154115436]
  We observed and interviewed ten experienced reviewers while they performed 25 code reviews from their review queue. Using Letovsky's model of code comprehension, we performed a theory-driven thematic analysis. Our findings confirm that code comprehension is fundamental to code review.
  arXiv  Detail & Related papers  (2025-03-27T12:44:40Z)
• Exploring the Effect of Explanation Content and Format on User Comprehension and Trust [11.433655064494896]
  We focus on explanations for a regression tool for assessing cancer risk. We examine the effect of the explanations' content and format on the user-centric metrics of comprehension and trust.
  arXiv  Detail & Related papers  (2024-08-30T16:36:53Z)
• Evaluating Evidence Attribution in Generated Fact Checking Explanations [48.776087871960584]
  We introduce a novel evaluation protocol, citation masking and recovery, to assess attribution quality in generated explanations. Experiments reveal that the best-performing LLMs still generate explanations with inaccurate attributions. Human-curated evidence is essential for generating better explanations.
  arXiv  Detail & Related papers  (2024-06-18T14:13:13Z)
• What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perception [53.4840989321394]
  We analyze the effect of rationales generated by QA models to support their answers. We present users with incorrect answers and corresponding rationales in various formats. We measure the effectiveness of this feedback in patching these rationales through in-context learning.
  arXiv  Detail & Related papers  (2023-11-16T04:26:32Z)
• Demystifying Code Snippets in Code Reviews: A Study of the OpenStack and Qt Communities and A Practitioner Survey [6.091233191627442]
  We conduct a mixed-methods study to mine information and knowledge related to code snippets in code reviews. The study results highlight that reviewers can provide code snippets in appropriate scenarios to meet developers' specific information needs in code reviews.
  arXiv  Detail & Related papers  (2023-07-26T17:49:19Z)
• Explanation Needs in App Reviews: Taxonomy and Automated Detection [2.545133021829296]
  We explore the need for explanation expressed by users in app reviews. We manually coded a set of 1,730 app reviews from 8 apps and derived a taxonomy of Explanation Needs. Our best classifier identifies Explanation Needs in 486 unseen reviews of 4 different apps with a weighted F-score of 86%.
  arXiv  Detail & Related papers  (2023-07-10T06:48:01Z)
• Exploring the Advances in Identifying Useful Code Review Comments [0.0]
  This paper reflects the evolution of research on the usefulness of code review comments. It examines papers that define the usefulness of code review comments, mine and annotate datasets, study developers' perceptions, analyze factors from different aspects, and use machine learning classifiers to automatically predict the usefulness of code review comments.
  arXiv  Detail & Related papers  (2023-07-03T00:41:20Z)
• Counterfactual Explainable Recommendation [22.590877963169103]
  We propose Counterfactual Explainable Recommendation (CountER), which takes the insights of counterfactual reasoning from causal inference for explainable recommendation. CountER seeks simple (low complexity) and effective (high strength) explanations for the model decision. Results show that our model generates more accurate and effective explanations than state-of-the-art explainable recommendation models.
  arXiv  Detail & Related papers  (2021-08-24T06:37:57Z)
• Can We Automate Scientific Reviewing? [89.50052670307434]
  We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
  arXiv  Detail & Related papers  (2021-01-30T07:16:53Z)
• Evaluating Explanations: How much do explanations from the teacher aid students? [103.05037537415811]
  We formalize the value of explanations using a student-teacher paradigm that measures the extent to which explanations improve student models in learning. Unlike many prior proposals to evaluate explanations, our approach cannot be easily gamed, enabling principled, scalable, and automatic evaluation of attributions.
  arXiv  Detail & Related papers  (2020-12-01T23:40:21Z)
• ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis [62.76038841302741]
  We build a novel ReviewRobot to automatically assign a review score and write comments for multiple categories such as novelty and meaningful comparison. Experimental results show that our review score predictor reaches 71.4%-100% accuracy. Human assessment by domain experts shows that 41.7%-70.5% of the comments generated by ReviewRobot are valid and constructive, and better than human-written ones for 20% of the time.
  arXiv  Detail & Related papers  (2020-10-13T02:17:58Z)
• Deep Just-In-Time Inconsistency Detection Between Comments and Source Code [51.00904399653609]
  In this paper, we aim to detect whether a comment becomes inconsistent as a result of changes to the corresponding body of code. We develop a deep-learning approach that learns to correlate a comment with code changes. We show the usefulness of our approach by combining it with a comment update model to build a more comprehensive automatic comment maintenance system.
  arXiv  Detail & Related papers  (2020-10-04T16:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.