FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes

• URL: http://arxiv.org/abs/2606.20769v1
• Date: Thu, 18 Jun 2026 15:06:36 GMT
• Title: FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes
• Abstract summary: We introduce FirstPass, a dataset fine-tuned model that addresses all three fronts of AI for peer review.<n>We exploit mandatory transparent peer review (instituted November 2022) and verify 100% content integrity by automated audit.<n>On generation, FirstPass produces reviews averaging 1,187 words, substantially closer to human references (2,155 words) than any baseline.
• Score: 0.27998963147546146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI systems for peer review fail on three fronts: they train on Computer Science and Machine Learning venues alone, ignore the iterative dialogue that validates science, and evaluate on stylistic mimicry rather than real editorial judgment. We introduce FirstPass, a dataset and fine-tuned model that addresses all three. Curating 3,668 complete multi-round peer-review dialogues from Nature Communications across five scientific domains (biology, chemistry, neuroscience, physics, and earth science), we exploit mandatory transparent peer review (instituted November 2022) and verify 100% content integrity by automated audit. We fine-tune Qwen2.5-7B-Instruct via Low-Rank Adaptation (LoRA) on three tasks: review generation, reviewer updating, and revision-cycle prediction. Our key finding is that response-only loss masking is a prerequisite, not an optimization: without it, accuracy is 62.0%, below the majority baseline; with it, FirstPass achieves 80.5% accuracy and F1-macro 78.2% on predicting editorial outcomes (Standard vs. Extended revision cycles), outperforming Gemini-3.1-flash-lite-preview zero-shot by 10.4 percentage points and all baselines with statistical significance (McNemar p < 0.001). On generation, FirstPass produces reviews averaging 1,187 words, substantially closer to human references (2,155 words) than any baseline, achieving ROUGE-L 0.154 with significant gains over Qwen and DeepSeek zero-shot (p < 0.001). Deployed in the pre-submission loop as an anticipatory scientific co-author, FirstPass simulates expert critique and predicts revision cycle outcomes before submission, giving authors the judgment a trusted colleague would provide, with consistent cross-domain performance across five disciplines.

Related papers

• Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review [2.9152186617858304]
  We study 11,429 reviews paired with 10,104 human-authored inline comments from AI.<n>Approval rates rise from 30.5% in reviewers' early periods to 36.6% in late periods.<n>Four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles.
  arXiv  Detail & Related papers  (2026-09-05T18:15:18Z)
• SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators [50.129606229339146]
  SurveyReview is a reviewer-aligned, multi-dimensional benchmark and dataset for survey evaluation.<n>We structure authentic peer-review reports by converting free-form comments into four-dimensional scores.<n>We release standardized train/test splits and an evaluation protocol to measure alignment between automatic evaluators and human reviewers.
  arXiv  Detail & Related papers  (2026-08-07T16:11:46Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• Benchmarking Agentic Review Systems [25.437512824197736]
  A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research.<n>We evaluate two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline.<n>We show that while AI reviews still have room for improvement, they can already track human quality judgments well, catch important errors, and earn positive feedback from real users.
  arXiv  Detail & Related papers  (2026-06-18T03:30:06Z)
• Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes [0.0]
  Large language model (LLM) systems are increasingly proposed to assist peer review.<n>Most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.<n>We validate AIPR, which reads a submitted manuscript and emits five 0-100 quality dimensions and a weighted overall score.
  arXiv  Detail & Related papers  (2026-06-14T16:13:15Z)
• On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists [113.03797263688519]
  Many scientists view AI reviewers as probabilistic systems without the expertise to evaluate research.<n>Existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts.
  arXiv  Detail & Related papers  (2026-05-20T03:33:55Z)
• The Compliance Gap: Why AI Systems Promise to Follow Process Instructions but Don't [0.0]
  We call this the Compliance Gap: a third, axis of AI honesty distinct from factual truthfulness and rhetorical substance.<n>Some 75 benchmarks measure outcome fidelity; none measures process fidelity.<n>We release BS-Bench: the first open benchmark for process compliance, with seven tool-call-log audit metrics and a public leaderboard.
  arXiv  Detail & Related papers  (2026-05-03T08:11:15Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs [29.26861081722613]
  We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics.<n>We introduce QEDBench, the first large-scale dual-rubric alignment benchmark to measure alignment with human experts on university-level math.<n>We reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias.
  arXiv  Detail & Related papers  (2026-02-24T07:23:28Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• DS@GT at CheckThat! 2025: Evaluating Context and Tokenization Strategies for Numerical Fact Verification [49.1574468325115]
  Numerical claims, statements involving quantities, comparisons, and temporal references pose unique challenges for automated fact-checking systems.<n>We evaluate modeling strategies for veracity prediction of such claims using the QuanTemp dataset and building our own evidence retrieval pipeline.<n>Our best-performing system achieves competitive macro-average F1 score of 0.57 and places us among the Top-4 submissions in Task 3 of CheckThat! 2025.
  arXiv  Detail & Related papers  (2025-07-08T17:22:22Z)
• News Summarization and Evaluation in the Era of GPT-3 [73.48220043216087]
  We study how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality.
  arXiv  Detail & Related papers  (2022-09-26T01:04:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.