From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

• URL: http://arxiv.org/abs/2608.27442v1
• Date: Thu, 27 Aug 2026 17:56:24 GMT
• Title: From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench
• Abstract summary: MCR-Bench is the first defect state-aware benchmark for realistic multi-round code review.<n>It consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information.
• Score: 47.66050051250628
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.

Related papers

• MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding [83.0805548642958]
  MTAC-IFBench is a comprehensive benchmark for instruction-following in software development.<n>It features multi-turn progressive software development instructions with diverse constraints spanning 6 primary and 18 secondary categories.<n>MTAC-IFBench identifies significant deficiencies in existing code agents in multi-turn instruction-following, with their performance degrading rapidly as the interaction session grows longer.
  arXiv  Detail & Related papers  (2026-09-14T03:57:37Z)
• MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs [16.970480774847857]
  We define Proactive Critique as the model's autonomous ability to identify, analyze and fix faulty user inputs without extra prompts.<n>We propose MMPCBench, a comprehensive framework for evaluating MLLMs' proactive critique competence.<n>It features a fine-grained taxonomy of 4 primary error types spanning 12 subcategories, ranging from cross-modal contradictions to missing visual premises.
  arXiv  Detail & Related papers  (2026-08-29T14:19:10Z)
• DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues [54.31138496553705]
  DiCoBench is a comprehensive, multi-image high-resolution benchmark for fine-grained perception.<n>Our evaluation reveals a striking performance gap compared to human accuracy (98.3%)<n>We believe DiCoBench will serve as a challenging testbed to drive future research in autonomous, high-resolution multi-image perception.
  arXiv  Detail & Related papers  (2026-06-25T05:02:38Z)
• Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation [59.87131391492088]
  We present Omni-I2C, a benchmark designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code.<n>By incorporating authentic user-sourced cases, the benchmark spans a vast spectrum of digital content.<n>Our evaluation reveals a substantial performance gap among leading LMMs.
  arXiv  Detail & Related papers  (2026-03-18T09:10:04Z)
• AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context [10.769682566098695]
  AACR-Bench is a comprehensive benchmark that provides full cross-file context across multiple programming languages.<n>Unlike traditional datasets, AACR-Bench employs an "AI-assisted, Expert-verified" annotation pipeline to uncover latent defects.
  arXiv  Detail & Related papers  (2026-01-27T11:28:44Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• Benchmarking Correctness and Security in Multi-Turn Code Generation [41.75392001830794]
  We introduce MTSec, the first benchmark to evaluate correctness and security in multi-turn coding scenarios.<n>We evaluate 32 open- and closed-source models, and three agent-scaffolding on MT-Sec.<n>We find that while agent-generated scaffoldings boost single-turn code generation performance, they are not quite as effective in multiturn evaluations.
  arXiv  Detail & Related papers  (2025-10-13T01:20:46Z)
• CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects [23.9752442213364]
  We introduce CodeFuse-CR-Bench, the first comprehensiveness-aware benchmark for repository-level CR evaluation.<n>CodeFuse-CR-Bench comprises 601 high-quality instances from 70 Python projects covering nine Pull-Request (PR) problem domains.<n>We present the first large-scale assessment of state-of-the-art Large Language Models (LLMs) on this comprehensive CR task.
  arXiv  Detail & Related papers  (2025-09-18T11:24:09Z)
• Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation [9.434966074326056]
  Multimodal Large Language Models (MLLMs) show impressive vision-language benchmark performance, yet growing concerns about data contamination risk masking true generalization.<n>We propose a novel dynamic evaluation framework to rigorously assess MLLM generalization, moving beyond static benchmarks.<n>We demonstrate that fine-tuning on simulated test data (extreme contamination) drastically sharpens task-specific performance but harms overall generalization.
  arXiv  Detail & Related papers  (2025-06-08T15:52:38Z)
• MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation [56.87891213797931]
  We present MTR-Bench for Large Language Models' Multi-Turn Reasoning evaluation.<n>Comprising 4 classes, 40 tasks, and 3600 instances, MTR-Bench covers diverse reasoning capabilities.<n>MTR-Bench features fully-automated framework spanning both dataset constructions and model evaluations.
  arXiv  Detail & Related papers  (2025-05-21T17:59:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.