On Assessing the Relevance of Code Reviews Authored by Generative Models

• URL: http://arxiv.org/abs/2512.15466v1
• Date: Wed, 17 Dec 2025 14:12:31 GMT
• Title: On Assessing the Relevance of Code Reviews Authored by Generative Models
• Authors: Robert Heumüller, Frank Ortmeier,
• Abstract summary: We propose a novel evaluation approach based on what we call multi-subjective ranking.<n>Using a dataset of 280 self-contained code review requests and corresponding comments from CodeReview StackExchange, multiple human judges ranked the quality of ChatGPT-generated comments alongside the top human responses from the platform.<n>Results show that ChatGPT's comments were ranked significantly better than human ones, even surpassing StackExchange's accepted answers.
• Score: 4.096540146408279
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The use of large language models like ChatGPT in code review offers promising efficiency gains but also raises concerns about correctness and safety. Existing evaluation methods for code review generation either rely on automatic comparisons to a single ground truth, which fails to capture the variability of human perspectives, or on subjective assessments of "usefulness", a highly ambiguous concept. We propose a novel evaluation approach based on what we call multi-subjective ranking. Using a dataset of 280 self-contained code review requests and corresponding comments from CodeReview StackExchange, multiple human judges ranked the quality of ChatGPT-generated comments alongside the top human responses from the platform. Results show that ChatGPT's comments were ranked significantly better than human ones, even surpassing StackExchange's accepted answers. Going further, our proposed method motivates and enables more meaningful assessments of generative AI's performance in code review, while also raising awareness of potential risks of unchecked integration into review processes.

Related papers

• Studying Quality Improvements Recommended via Manual and Automated Code Review [14.067404766521607]
  We study the similarities and differences between code reviews performed by humans and those automatically generated by Deep Learning models.<n>We show that while ChatGPT tends to recommend a higher number of code changes as compared to human reviewers, it can only spot 10% of the quality issues reported by humans.<n>This finding suggests that, in its current state, DL-based code review can be used as a further quality check on top of the one performed by humans.
  arXiv  Detail & Related papers  (2026-02-12T13:23:43Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• Leveraging Reward Models for Guiding Code Review Comment Generation [13.306560805316103]
  Code review is a crucial component of modern software development, involving the evaluation of code quality, providing feedback on potential issues, and refining the code to address identified problems.<n>Deep learning techniques are able to tackle the generative aspect of code review, by commenting on a given code as a human reviewer would do.<n>In this paper, we introduce CoRAL, a deep learning framework automating review comment generation by exploiting reinforcement learning with a reward mechanism.
  arXiv  Detail & Related papers  (2025-06-04T21:31:38Z)
• CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models [97.18215355266143]
  We introduce a holistic code critique benchmark for Large Language Models (LLMs) called CodeCriticBench.<n>Specifically, our CodeCriticBench includes two mainstream code tasks (i.e., code generation and code QA) with different difficulties.<n>Besides, the evaluation protocols include basic critique evaluation and advanced critique evaluation for different characteristics.
  arXiv  Detail & Related papers  (2025-02-23T15:36:43Z)
• Deep Assessment of Code Review Generation Approaches: Beyond Lexical Similarity [27.92468098611616]
  We propose two novel semantic-based approaches for assessing code reviews.<n>The first approach involves converting both the generated review and its reference into digital vectors using a deep learning model.<n>The second approach generates a prompt based on the generated review and its reference, submits this prompt to ChatGPT, and requests ChatGPT to rate the generated review.
  arXiv  Detail & Related papers  (2025-01-09T11:52:32Z)
• HREF: Human Response-Guided Evaluation of Instruction Following in Language Models [61.273153125847166]
  We develop a new evaluation benchmark, Human Response-Guided Evaluation of Instruction Following (HREF)<n>In addition to providing reliable evaluation, HREF emphasizes individual task performance and is free from contamination.<n>We study the impact of key design choices in HREF, including the size of the evaluation set, the judge model, the baseline model, and the prompt template.
  arXiv  Detail & Related papers  (2024-12-20T03:26:47Z)
• CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells [15.66562304661042]
  CRScore is a reference-free metric to measure dimensions of review quality like conciseness, comprehensiveness, and relevance.<n>We demonstrate that CRScore can produce valid, fine-grained scores of review quality that have the greatest alignment with human judgment among open source metrics.<n>We also release a corpus of 2.9k human-annotated review quality scores for machine-generated and GitHub review comments to support the development of automated metrics.
  arXiv  Detail & Related papers  (2024-09-29T21:53:18Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study [0.0]
  ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks. We conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks. Our results show that ChatGPT achieves higher EM and BLEU scores of 22.78 and 76.44 respectively, while the state-of-the-art method achieves only 15.50 and 62.88 on a high-quality code review dataset.
  arXiv  Detail & Related papers  (2023-09-15T07:41:33Z)
• Large Language Models are not Fair Evaluators [60.27164804083752]
  We find that the quality ranking of candidate responses can be easily hacked by altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other. We propose a framework with three simple yet effective strategies to mitigate this issue.
  arXiv  Detail & Related papers  (2023-05-29T07:41:03Z)
• Deep Just-In-Time Inconsistency Detection Between Comments and Source Code [51.00904399653609]
  In this paper, we aim to detect whether a comment becomes inconsistent as a result of changes to the corresponding body of code. We develop a deep-learning approach that learns to correlate a comment with code changes. We show the usefulness of our approach by combining it with a comment update model to build a more comprehensive automatic comment maintenance system.
  arXiv  Detail & Related papers  (2020-10-04T16:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.