BitsAI-CR: Automated Code Review via LLM in Practice

• URL: http://arxiv.org/abs/2501.15134v1
• Date: Sat, 25 Jan 2025 08:39:50 GMT
• Title: BitsAI-CR: Automated Code Review via LLM in Practice
• Authors: Tao Sun, Jian Xu, Yuanpeng Li, Zhao Yan, Ge Zhang, Lintao Xie, Lu Geng, Zheng Wang, Yueyan Chen, Qin Lin, Wenbo Duan, Kaixin Sui,
• Abstract summary: BitsAI-CR is an innovative framework that enhances code review through a two-stage approach.<n>System is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism.<n> Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation.
• Score: 16.569842114384233
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code review remains a critical yet resource-intensive process in software development, particularly challenging in large-scale industrial environments. While Large Language Models (LLMs) show promise for automating code review, existing solutions face significant limitations in precision and practicality. This paper presents BitsAI-CR, an innovative framework that enhances code review through a two-stage approach combining RuleChecker for initial issue detection and ReviewFilter for precision verification. The system is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism that enables continuous performance improvement through structured feedback and evaluation metrics. Our approach introduces an Outdated Rate metric that can reflect developers' actual adoption of review comments, enabling automated evaluation and systematic optimization at scale. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation. For the Go language which has predominant usage at ByteDance, we maintain an Outdated Rate of 26.7%. The system has been successfully deployed at ByteDance, serving over 12,000 Weekly Active Users (WAU). Our work provides valuable insights into the practical application of automated code review and offers a blueprint for organizations seeking to implement automated code reviews at scale.

Related papers

• The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models [53.12387628636912]
  We propose an automatic evaluation framework that is validated against human annotations. This approach was originally developed for the TREC Question Answering (QA) Track in 2003. We observe strong agreement at the run level between scores derived from fully automatic nugget evaluation and human-based variants.
  arXiv  Detail & Related papers  (2025-04-21T12:55:06Z)
• Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute [61.00662702026523]
  We propose a unified Test-Time Compute scaling framework that leverages increased inference-time instead of larger models. Our framework incorporates two complementary strategies: internal TTC and external TTC. We demonstrate our textbf32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1.
  arXiv  Detail & Related papers  (2025-03-31T07:31:32Z)
• CodeArena: A Collective Evaluation Platform for LLM Code Generation [46.800918377886184]
  CodeArena is an online evaluation framework tailored for Large Language Models (LLMs) code generation. The key innovation is a collective evaluation mechanism, which recalibrates individual model scores based on the holistic performance of all participating models. CodeArena ensures open access to all submitted solutions and test cases and provides automation-friendly APIs to streamline the code evaluation workflow.
  arXiv  Detail & Related papers  (2025-03-03T08:31:16Z)
• Harnessing Large Language Models for Curated Code Reviews [2.5944208050492183]
  In code review, generating structured and relevant comments is crucial for identifying code issues and facilitating accurate code changes. Existing code review datasets are often noisy and unrefined, posing limitations to the learning potential of AI models. We propose a curation pipeline designed to enhance the quality of the largest publicly available code review dataset.
  arXiv  Detail & Related papers  (2025-02-05T18:15:09Z)
• Deep Assessment of Code Review Generation Approaches: Beyond Lexical Similarity [27.92468098611616]
  We propose two novel semantic-based approaches for assessing code reviews.<n>The first approach involves converting both the generated review and its reference into digital vectors using a deep learning model.<n>The second approach generates a prompt based on the generated review and its reference, submits this prompt to ChatGPT, and requests ChatGPT to rate the generated review.
  arXiv  Detail & Related papers  (2025-01-09T11:52:32Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)
• Predicting Expert Evaluations in Software Code Reviews [8.012861163935904]
  This paper presents an algorithmic model that automates aspects of code review typically avoided due to their complexity or subjectivity. Instead of replacing manual reviews, our model adds insights that help reviewers focus on more impactful tasks.
  arXiv  Detail & Related papers  (2024-09-23T16:01:52Z)
• AI-Assisted Assessment of Coding Practices in Modern Code Review [11.803776132972029]
  AutoCommenter is an end-to-end system for learning and enforcing coding best practices. This paper reports on the development, deployment, and evaluation of AutoCommenter.
  arXiv  Detail & Related papers  (2024-05-22T11:57:18Z)
• Automating Patch Set Generation from Code Review Comments Using Large Language Models [2.045040820541428]
  We provide code contexts to five popular Large Language Models (LLMs) We obtain the suggested code-changes (patch sets) derived from real-world code-review comments. The performance of each model is meticulously assessed by comparing their generated patch sets against the historical data of human-generated patch-sets.
  arXiv  Detail & Related papers  (2024-04-10T02:46:08Z)
• The Right Prompts for the Job: Repair Code-Review Defects with Large Language Model [15.885824575879763]
  Automatic program repair (APR) techniques have the potential to reduce manual efforts in uncovering and repairing program defects during the code review (CR) process. However, the limited accuracy and considerable time costs associated with existing APR approaches hinder their adoption in industrial practice. Recent advancements in Large Language Models (LLMs) have enhanced their ability to comprehend natural and programming languages, enabling them to generate patches based on review comments.
  arXiv  Detail & Related papers  (2023-12-29T06:12:15Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework [91.94389491920309]
  Textual adversarial attacks can discover models' weaknesses by adding semantic-preserved but misleading perturbations to the inputs. The existing practice of robustness evaluation may exhibit issues of incomprehensive evaluation, impractical evaluation protocol, and invalid adversarial samples. We set up a unified automatic robustness evaluation framework, shifting towards model-centric evaluation to exploit the advantages of adversarial attacks.
  arXiv  Detail & Related papers  (2023-05-29T14:55:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.