BitsAI-CR: Automated Code Review via LLM in Practice

• URL: http://arxiv.org/abs/2501.15134v1
• Date: Sat, 25 Jan 2025 08:39:50 GMT
• Title: BitsAI-CR: Automated Code Review via LLM in Practice
• Authors: Tao Sun, Jian Xu, Yuanpeng Li, Zhao Yan, Ge Zhang, Lintao Xie, Lu Geng, Zheng Wang, Yueyan Chen, Qin Lin, Wenbo Duan, Kaixin Sui,
• Abstract summary: BitsAI-CR is an innovative framework that enhances code review through a two-stage approach. System is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation.
• Score: 16.569842114384233
• License:
• Abstract: Code review remains a critical yet resource-intensive process in software development, particularly challenging in large-scale industrial environments. While Large Language Models (LLMs) show promise for automating code review, existing solutions face significant limitations in precision and practicality. This paper presents BitsAI-CR, an innovative framework that enhances code review through a two-stage approach combining RuleChecker for initial issue detection and ReviewFilter for precision verification. The system is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism that enables continuous performance improvement through structured feedback and evaluation metrics. Our approach introduces an Outdated Rate metric that can reflect developers' actual adoption of review comments, enabling automated evaluation and systematic optimization at scale. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation. For the Go language which has predominant usage at ByteDance, we maintain an Outdated Rate of 26.7%. The system has been successfully deployed at ByteDance, serving over 12,000 Weekly Active Users (WAU). Our work provides valuable insights into the practical application of automated code review and offers a blueprint for organizations seeking to implement automated code reviews at scale.

Related papers

• Scoring Verifiers: Evaluating Synthetic Verification in Code and Reasoning [59.25951947621526]
  We introduce benchmarks designed to evaluate the impact of synthetic verification methods on assessing solution correctness. We analyze synthetic verification methods in standard, reasoning-based, and reward-based LLMs. Our results show that recent reasoning models significantly improve test case generation and that scaling test cases enhances verification accuracy.
  arXiv  Detail & Related papers  (2025-02-19T15:32:11Z)
• Harnessing Large Language Models for Curated Code Reviews [2.5944208050492183]
  In code review, generating structured and relevant comments is crucial for identifying code issues and facilitating accurate code changes. Existing code review datasets are often noisy and unrefined, posing limitations to the learning potential of AI models. We propose a curation pipeline designed to enhance the quality of the largest publicly available code review dataset.
  arXiv  Detail & Related papers  (2025-02-05T18:15:09Z)
• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain [62.89809156574998]
  We introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain. Our benchmark is characterized by its multi-dimensional evaluation framework. Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets.
  arXiv  Detail & Related papers  (2024-12-17T15:38:42Z)
• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)
• Predicting Expert Evaluations in Software Code Reviews [8.012861163935904]
  This paper presents an algorithmic model that automates aspects of code review typically avoided due to their complexity or subjectivity. Instead of replacing manual reviews, our model adds insights that help reviewers focus on more impactful tasks.
  arXiv  Detail & Related papers  (2024-09-23T16:01:52Z)
• AI-Assisted Assessment of Coding Practices in Modern Code Review [11.803776132972029]
  AutoCommenter is an end-to-end system for learning and enforcing coding best practices. This paper reports on the development, deployment, and evaluation of AutoCommenter.
  arXiv  Detail & Related papers  (2024-05-22T11:57:18Z)
• Automating Patch Set Generation from Code Review Comments Using Large Language Models [2.045040820541428]
  We provide code contexts to five popular Large Language Models (LLMs) We obtain the suggested code-changes (patch sets) derived from real-world code-review comments. The performance of each model is meticulously assessed by comparing their generated patch sets against the historical data of human-generated patch-sets.
  arXiv  Detail & Related papers  (2024-04-10T02:46:08Z)
• The Right Prompts for the Job: Repair Code-Review Defects with Large Language Model [15.885824575879763]
  Automatic program repair (APR) techniques have the potential to reduce manual efforts in uncovering and repairing program defects during the code review (CR) process. However, the limited accuracy and considerable time costs associated with existing APR approaches hinder their adoption in industrial practice. Recent advancements in Large Language Models (LLMs) have enhanced their ability to comprehend natural and programming languages, enabling them to generate patches based on review comments.
  arXiv  Detail & Related papers  (2023-12-29T06:12:15Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework [91.94389491920309]
  Textual adversarial attacks can discover models' weaknesses by adding semantic-preserved but misleading perturbations to the inputs. The existing practice of robustness evaluation may exhibit issues of incomprehensive evaluation, impractical evaluation protocol, and invalid adversarial samples. We set up a unified automatic robustness evaluation framework, shifting towards model-centric evaluation to exploit the advantages of adversarial attacks.
  arXiv  Detail & Related papers  (2023-05-29T14:55:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.