Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning

• URL: http://arxiv.org/abs/2607.15388v1
• Date: Thu, 16 Jul 2026 18:38:10 GMT
• Title: Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning
• Abstract summary: We test the assumption that a dedicated review stage should help turn wrong candidates into correct ones.<n>We ask whether this gap is explained by reviewer quality or by whether critique changes the next answer the protocol carries forward.
• Score: 9.345403722854543
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many math- and science-oriented agent systems use hierarchical designs with specialized reviewer roles, assuming that a dedicated review stage should help turn wrong candidates into correct ones. We test this assumption on 4,181 verifier-grounded Omni-MATH problems using matched gpt-oss-120b actors. Collaboration adds little on the easiest tiers, but from tier 4 onward the gains open sharply; in this harder regime, broadcast-style peer discussion reaches higher final accuracy than a planner-executor-reviewer pipeline (PER). We ask whether this gap is explained by reviewer quality or by whether critique changes the next answer the protocol carries forward. It is not explained by reviewer precision alone: PER's reviewer is more precise than broadcast's (0.861 vs. 0.644), yet evaluator-verified useful critique is much less likely to change the next candidate and produces lower reviewer-guided repair. These results show that reviewer detection quality and critique uptake are empirically separable. Within matched PER interventions, forcing explicit acknowledgment lowers final accuracy, while embedding reviewer guidance directly in the solver's working context partially improves follow-through without closing the gap. Overall, reviewer-centric evaluation can overstate system quality: a protocol may spot errors well yet still fail to solve more problems if it does not act on those critiques.

Related papers

• Reviewer Capability Governs Rejection Targeting, Not Repair Skill: Evidence from LLM Execute-Review-Revise Pipelines [0.0]
  Multi-agent LLM pipelines increasingly assign roles, including execution and verification, to models of different capability tiers.<n>Previous literature has established that verification stages are not always beneficial, but holds reviewer capability roughly fixed relative to the executor.<n>We replace the reviewer with models spanning a capability range down to one that cannot solve the problems at all, and measure the outcome of every individual rejection.
  arXiv  Detail & Related papers  (2026-09-02T19:21:49Z)
• Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Reviews [37.684628465527545]
  We operationalise Kahneman's dual-process theory into a structured rubric for peer review and release Kahneman4Review.<n>A benchmark of 3,563 rated reviews scored along nine theoretically motivated textual dimensions, eight bias diagnostics, and a continuous reasoning-quality score.
  arXiv  Detail & Related papers  (2026-07-12T00:03:42Z)
• SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review [47.72392142544407]
  Coding agents increasingly generate pull requests (PRs) for real-world software issues.<n>One-shot PR generation remains open-loop: the PR is proposed without systematic review, diagnosis, or revision.<n>We introduce textbfSWE-Review, a framework for closing this loop with agentic code review.
  arXiv  Detail & Related papers  (2026-07-07T09:37:45Z)
• Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes [0.0]
  Large language model (LLM) systems are increasingly proposed to assist peer review.<n>Most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.<n>We validate AIPR, which reads a submitted manuscript and emits five 0-100 quality dimensions and a weighted overall score.
  arXiv  Detail & Related papers  (2026-06-14T16:13:15Z)
• CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers [39.97153044457166]
  We build category-specific benchmark subsets and skip evaluation when human reviews are missing to strengthen Completeness.<n>We also leverage reviewer--author--meta-review discussions as expert annotations and filter unreliable reviews accordingly to strengthen Correctness.<n>CoCoReviewBench curates 3,900 papers from ICLR and NeurIPS to enable reliable and fine-grained evaluation of AI reviewers.
  arXiv  Detail & Related papers  (2026-05-08T15:44:26Z)
• Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents [6.158612515104146]
  We introduce Helpfulness-Harmfulness metrics to measure the tradeoff between agent feedback and reviewer feedback.<n>We evaluate our approach on BFCL and Tau2-Bench (multi-turn stateful scenarios), achieving +5.5% on irrelevance detection and +7.1% on multi-turn tasks.<n>Our metrics reveal that reviewer model choice is critical: the reasoning model o3-mini achieves a 3:1 benefit-to-risk ratio versus 2.1:1 for GPT-4o.
  arXiv  Detail & Related papers  (2026-04-29T22:09:47Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Reviewing the Reviewer: Elevating Peer Review Quality through LLM-Guided Feedback [75.31379834079648]
  We introduce an LLM-driven framework that decomposes reviews into argumentative segments.<n>We also release LazyReviewPlus, a dataset of 1,309 sentences labeled for lazy thinking and specificity.
  arXiv  Detail & Related papers  (2026-01-17T20:32:18Z)
• LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews [74.87393214734114]
  This work introduces LazyReview, a dataset of peer-review sentences annotated with fine-grained lazy thinking categories.<n>Large Language Models (LLMs) struggle to detect these instances in a zero-shot setting.<n> instruction-based fine-tuning on our dataset significantly boosts performance by 10-20 performance points.
  arXiv  Detail & Related papers  (2025-04-15T10:07:33Z)
• Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review [4.35783648216893]
  Traditional closed peer review systems are slow, costly, non-transparent, and possibly subject to biases.<n>We propose and examine the efficacy and accuracy of an alternative form of scientific peer review: through an open, bottom-up process.
  arXiv  Detail & Related papers  (2025-01-22T17:00:27Z)
• Large Language Models are not Fair Evaluators [60.27164804083752]
  We find that the quality ranking of candidate responses can be easily hacked by altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other. We propose a framework with three simple yet effective strategies to mitigate this issue.
  arXiv  Detail & Related papers  (2023-05-29T07:41:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.