CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
- URL: http://arxiv.org/abs/2605.07905v1
- Date: Fri, 08 May 2026 15:44:26 GMT
- Title: CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
- Abstract summary: We build category-specific benchmark subsets and skip evaluation when human reviews are missing to strengthen Completeness.<n>We also leverage reviewer--author--meta-review discussions as expert annotations and filter unreliable reviews accordingly to strengthen Correctness.<n>CoCoReviewBench curates 3,900 papers from ICLR and NeurIPS to enable reliable and fine-grained evaluation of AI reviewers.
- Score: 39.97153044457166
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Despite the rapid development of AI reviewers, evaluating such systems remains challenging: metrics favor overlap with human reviews over correctness. However, since human reviews often cover only a subset of salient issues and sometimes contain mistakes, they are unreliable as gold references. To address this, we build category-specific benchmark subsets and skip evaluation when the corresponding human reviews are missing to strengthen Completeness. We also leverage reviewer--author--meta-review discussions as expert annotations and filter unreliable reviews accordingly to strengthen Correctness. Finally, we introduce CoCoReviewBench, which curates 3,900 papers from ICLR and NeurIPS to enable reliable and fine-grained evaluation of AI reviewers. Analysis shows that AI reviewers remain limited in correctness and are prone to hallucinations, and highlights reasoning models as more effective reviewers, motivating further directions for improving AI reviewers. Benchmarks and models are available at https://github.com/hexuandeng/CoCoReviewBench.
Related papers
- AppliedScientist: Automated Scientific Revision Through Iterative AI Reviewing [10.712519962597604]
We present AppliedScientist, a closed-loop system that couples an autonomous AI scientist with an AI reviewer.<n>We evaluate it by iteratively revising rejected papers from a range of research subfields.
arXiv Detail & Related papers (2026-09-13T18:59:10Z) - ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation [54.82704239643649]
We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans.<n>Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback.<n>We also introduce ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness.
arXiv Detail & Related papers (2026-09-08T17:30:16Z) - When AI reviews science: Can we trust the referee? [73.47745294608072]
We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
arXiv Detail & Related papers (2026-04-26T08:03:32Z) - Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
arXiv Detail & Related papers (2026-04-21T14:21:15Z) - ReviewScore: Misinformed Peer Review Detection with Large Language Models [38.92827930465428]
We show that 15.2% of weaknesses and 26.4% of questions are misinformed and introduce ReviewScore indicating if a review point is misinformed.<n>We build a human expert-annotated ReviewScore dataset to check the ability of LLMs to automate ReviewScore evaluation.<n>We also prove that evaluating premise-level factuality shows significantly higher agreements than evaluating weakness-level factuality.
arXiv Detail & Related papers (2025-09-25T22:55:05Z) - The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors [45.98233565214142]
We identify four key aspects of review comments that drive the utility for authors: Actionability, Grounding & Specificity, Verifiability, and Helpfulness.<n>We collect 1,430 human-labeled review comments and scale our data with 10k synthetically labeled comments for training purposes.<n>We benchmark fine-tuned models for assessing review comments on these aspects and generating rationales.
arXiv Detail & Related papers (2025-08-31T14:19:07Z) - Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 [115.86204862475864]
Review Feedback Agent provides automated feedback on vague comments, content misunderstandings, and unprofessional remarks to reviewers.<n>It was implemented at ICLR 2025 as a large randomized control study.<n> 27% of reviewers who received feedback updated their reviews, and over 12,000 feedback suggestions from the agent were incorporated by those reviewers.
arXiv Detail & Related papers (2025-04-13T22:01:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.