More Criticism Does Not Make a Better Review: EquiReview-R

• URL: http://arxiv.org/abs/2609.03943v1
• Date: Thu, 03 Sep 2026 14:49:24 GMT
• Title: More Criticism Does Not Make a Better Review: EquiReview-R
• Abstract summary: We recast AI-assisted review as evidence-guided refinement of a structured concern set.<n>We introduce EquiReview-R, which resolves existing concerns against localized evidence.<n>On a frozen cohort of previously unseen papers, EquiReview-R satisfies the prespecified non-inferiority criterion for major omission.
• Score: 5.8331781561663645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI reviewers can now produce many specific criticisms, but more criticism is not necessarily a better review. A review may miss a consequential weakness or retain an allegation that available evidence does not support. These failures require opposite corrections, yet generation-oriented systems and aggregate measures obscure the distinction. We therefore recast AI-assisted review as evidence-guided refinement of a structured concern set, with omission and overcritique treated as separate risks. Building on this formulation, we introduce EquiReview-R, which resolves existing concerns against localized evidence, searches for missing issues from independent and review-conditioned perspectives, and returns stop, continue, or defer. To expose the failure mode that motivates this design, we construct an evidence-linked trajectory corpus. Its retrospective analysis shows why revision must precede further search: nearly all concerns in a high-recall review lack a definitive evidential disposition, while an earlier refinement mechanism cannot revise them. On a frozen cohort of previously unseen papers, EquiReview-R satisfies the prespecified non-inferiority criterion for major omission, reduces major overcritique from 15.5% to 8.1%, and attains a one-sided omission upper bound of 9.9% while stopping on 52.4% of papers. Computation-matched controls, controlled pairs, and ablations show that the gain comes from revision rather than extra inference or shorter output. We release the corpus as ReviewTrace, an evidence-linked resource for studying review revision, disagreement, and provenance.

Related papers

• AppliedScientist: Automated Scientific Revision Through Iterative AI Reviewing [10.712519962597604]
  We present AppliedScientist, a closed-loop system that couples an autonomous AI scientist with an AI reviewer.<n>We evaluate it by iteratively revising rejected papers from a range of research subfields.
  arXiv  Detail & Related papers  (2026-09-13T18:59:10Z)
• ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation [54.82704239643649]
  We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans.<n>Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback.<n>We also introduce ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness.
  arXiv  Detail & Related papers  (2026-09-08T17:30:16Z)
• AutoSupervision: Closing the Feedback Loop in Scientific Workflows with Grounded Revision Verification [26.43951827696846]
  We introduce AutoSupervision, which evaluates whether scientific manuscript revisions genuinely address reviewer concerns.<n>We construct AutoSupervision from 56,000 Nature Communications articles and corresponding review records.<n>Our results show that while LLMs perform well in characterizing reviewer concerns, evidence-based verification remains the primary bottleneck.
  arXiv  Detail & Related papers  (2026-07-30T08:24:58Z)
• Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning [9.345403722854543]
  We test the assumption that a dedicated review stage should help turn wrong candidates into correct ones.<n>We ask whether this gap is explained by reviewer quality or by whether critique changes the next answer the protocol carries forward.
  arXiv  Detail & Related papers  (2026-07-16T18:38:10Z)
• From Passive Generation to Investigation: A Proactive Scientific Peer Review Agent [52.42081442204093]
  We argue that a key limitation is the lack of flexibility to proactively investigate suspicious parts of a paper.<n>We propose ProReviewer, a scientific peer review agent that proactively reviews a paper guided by a maintained, structured review log.<n> Experiments show that ProReviewer with an 8B backbone, trained by supervised fine-tuning and optimized by reinforcement learning, achieves the highest average score across five quality dimensions.
  arXiv  Detail & Related papers  (2026-06-11T13:38:23Z)
• CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers [39.97153044457166]
  We build category-specific benchmark subsets and skip evaluation when human reviews are missing to strengthen Completeness.<n>We also leverage reviewer--author--meta-review discussions as expert annotations and filter unreliable reviews accordingly to strengthen Correctness.<n>CoCoReviewBench curates 3,900 papers from ICLR and NeurIPS to enable reliable and fine-grained evaluation of AI reviewers.
  arXiv  Detail & Related papers  (2026-05-08T15:44:26Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Review [6.59569431190247]
  We propose a diagnostic framework that evaluates AI reviews at the concern level rather than only at the verdict level.<n>We derive an evaluation ladder that moves from binary accuracy to concern detection, verdict-stratified behavior, decision-aware calibration, and rebuttal-aware decomposition.
  arXiv  Detail & Related papers  (2026-04-21T21:16:59Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification [57.196748998757954]
  We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification.<n>Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository.<n>It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive.
  arXiv  Detail & Related papers  (2026-04-05T11:45:22Z)
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
  We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
  arXiv  Detail & Related papers  (2025-08-29T08:48:00Z)
• How Useful are Reviews for Recommendation? A Critical Review and Potential Improvements [8.471274313213092]
  We investigate a growing body of work that seeks to improve recommender systems through the use of review text. Our initial findings reveal several discrepancies in reported results, partly due to copying results across papers despite changes in experimental settings or data pre-processing. Further investigation calls for discussion on a much larger problem about the "importance" of user reviews for recommendation.
  arXiv  Detail & Related papers  (2020-05-25T16:30:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.