Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes

• URL: http://arxiv.org/abs/2606.15887v1
• Date: Sun, 14 Jun 2026 16:13:15 GMT
• Title: Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes
• Abstract summary: Large language model (LLM) systems are increasingly proposed to assist peer review.<n>Most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.<n>We validate AIPR, which reads a submitted manuscript and emits five 0-100 quality dimensions and a weighted overall score.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns. We validate AIPR, which reads a submitted manuscript and emits five 0-100 quality dimensions and a weighted overall score, against the public decision outcomes of a major machine learning venue. AIPR grades by prompting alone, with no fine-tuning on reviews or decisions. Across 300 ICLR submissions with public decision tiers and reviewer ratings, graded under a frozen pipeline with hypotheses pre-registered before any score met any outcome, the overall score separates rejected from accepted submissions (AUROC 0.82, 95% CI 0.78-0.87), rises monotonically across tiers, and tracks the mean reviewer rating. The signal is strongest where we claim it: the lowest-scoring fifth is rejected far above the base rate, with oral papers absent. The validity comes mostly from the model: a one-paragraph prompt on the same model discriminates almost as well as the full pipeline (the small gap favours the pipeline but does not meet the pre-declared criterion, p = 0.09). What the engineering adds is reliability and a grounded review: AIPR's score barely moves across repeated runs (0.7 vs. 2.8 points within-paper SD) where the bare prompt swings, and the same pass returns a rubric-structured, evidence-grounded review rather than a bare number, with the human keeping the decision.

Related papers

• WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays [16.22466124017634]
  WrAFT is a writing assessment and feedback tool for argumentative essays.<n>System achieves state-of-the-art performance in scoring, with a quadratic weighted kappa (QWK) of 0.84 and a root mean square error (RMSE) of 0.44 against official scores on a scale of 0-5.<n>An interactive user interface has been developed for the system and is publicly available and free to use.
  arXiv  Detail & Related papers  (2026-07-16T03:23:31Z)
• FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes [0.27998963147546146]
  We introduce FirstPass, a dataset fine-tuned model that addresses all three fronts of AI for peer review.<n>We exploit mandatory transparent peer review (instituted November 2022) and verify 100% content integrity by automated audit.<n>On generation, FirstPass produces reviews averaging 1,187 words, substantially closer to human references (2,155 words) than any baseline.
  arXiv  Detail & Related papers  (2026-06-18T15:06:36Z)
• Benchmarking Agentic Review Systems [25.437512824197736]
  A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research.<n>We evaluate two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline.<n>We show that while AI reviews still have room for improvement, they can already track human quality judgments well, catch important errors, and earn positive feedback from real users.
  arXiv  Detail & Related papers  (2026-06-18T03:30:06Z)
• A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR [1.2958054117511815]
  Reinforcement learning from rewards improves reasoning even when the reward signal is spurious.<n> Practitioners commonly interpret naive = acc(TRUE) - acc(R) as the reward-design effect.<n>We prove this estimand is systematically biased.
  arXiv  Detail & Related papers  (2026-06-04T09:35:54Z)
• SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones? [51.154921661608675]
  We introduce SoundnessBench, a curated benchmark of 1,099 machine-learning research proposals reconstructed from ICLR submissions.<n>SoundnessBench should be interpreted as a benchmark for recoverable proposal-stage soundness rather than exact prediction of full-paper review outcomes.
  arXiv  Detail & Related papers  (2026-05-28T17:57:37Z)
• VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference [0.0]
  VERDI (VERification-Decomposed Inference Inference) is a method that extracts confidence from a structured judge.<n> VERDI decomposes each evaluation into sub-checks and derives three structural signals.<n>On three public benchmarks, VERDI achieves AUROC 0.72-0.91 on GPT-4.1-mini and 0.66-0.80 on GPT-5.4-mini.
  arXiv  Detail & Related papers  (2026-05-11T23:39:19Z)
• Stop Automating Peer Review Without Rigorous Evaluation [51.53099943385505]
  This position paper argues that today's AI systems should not be used to produce paper reviews.<n>We identify two critical issues: 1) AI reviewers exhibit a hivemind effect of excessive agreement within and across papers that reduces perspective diversity.<n>We argue that addressing the peer review crisis requires a science of peer review automation.
  arXiv  Detail & Related papers  (2026-05-04T22:41:04Z)
• Reviewer Scores Are Not Comparable Across Research Areas in ML Peer Review [5.466519893996914]
  This position paper argues that acceptance outcomes are shaped by forces beyond reviewer scores.<n>We show that at any given reviewer score, a paper's acceptance probability varies by up to 8x depending on its topic.<n>We call on program committees to adopt inherently incomparable review signals and publish topic-stratified, score-conditional acceptance rates.
  arXiv  Detail & Related papers  (2026-04-30T10:54:54Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• Decoupling Scores and Text: The Politeness Principle in Peer Review [0.0]
  Authors often struggle to interpret peer review feedback, deriving false hope from polite comments or feeling confused by specific low scores.<n>We construct a dataset of over 30,000 ICLR 2021-2025 submissions and compare acceptance prediction performance using numerical scores versus text reviews.<n>Our experiments reveal a significant performance gap: score-based models achieve 91% accuracy, while text-based models reach only 81% even with large language models.
  arXiv  Detail & Related papers  (2026-03-23T11:58:48Z)
• Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation [3.23600523782706]
  We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning.<n>We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which penalizes disordered reasoning chains.<n>CPR-Curriculum achieves a 32% improvement in Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.
  arXiv  Detail & Related papers  (2026-03-13T15:48:15Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning [64.46921169261852]
  RAG-Zeval is a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task.<n>Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments.<n>Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments.
  arXiv  Detail & Related papers  (2025-05-28T14:55:33Z)
• Integrating Rankings into Quantized Scores in Peer Review [61.27794774537103]
  In peer review, reviewers are usually asked to provide scores for the papers. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are no standard procedure for using this ranking information and Area Chairs may use it in different ways. We take a principled approach to integrate the ranking information into the scores.
  arXiv  Detail & Related papers  (2022-04-05T19:39:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.