Local AI pre-screening for human triple-blind peer review in health sciences

• URL: http://arxiv.org/abs/2608.14625v1
• Date: Sat, 18 Jul 2026 00:06:33 GMT
• Title: Local AI pre-screening for human triple-blind peer review in health sciences
• Abstract summary: An analysis of ICLR 2026 found roughly 21% of its 75,800 peer reviews were fully AI-generated.<n>We propose a triple-blind, multi-LLM pre-screening framework for peer review.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Academic peer review is under mounting strain: NeurIPS 2025 received 21,575 submissions, ICLR 2025 received 11,603, and ICML 2025 received 12,107. This volume has outpaced the supply of qualified reviewers, and large language models (LLMs) are already filling the gap, largely undisclosed. An independent analysis of ICLR 2026 found roughly 21% of its 75,800 peer reviews were fully AI-generated, with over half showing some AI involvement (up from 15.8% in 2024). Documented risks include hallucinated citations in accepted papers and hidden prompt-injection instructions embedded in manuscripts to manipulate AI reviewers into favorable assessments. We propose a triple-blind, multi-LLM pre-screening framework for peer review, developed for a health sciences journal, that formalizes and discloses AI involvement while preserving human reviewers as the final decision-making authority. The framework routes a submission through five stages -- sanitization/anonymization, parallel AI pre-screening, an automated check gate, blinded human review, and editorial adjudication -- with return-to-author loops at the check and editor stages. Addressing the confidentiality concerns behind NIH/NSF bans on submitting unpublished proposals to third-party generative AI, all three AI reviewers run on locally-hosted, open-weight LLMs, keeping manuscript content within the journal infrastructure. The closest precedent, Shen et al., benchmarked five open-source LLMs on quartile classification of 200 manuscripts and found accuracy insufficient (35% exact-match) for autonomous use, supporting our decision to retain mandatory human adjudication. This transparent, human-supervised design offers a defensible alternative to today's opaque, unregulated AI use in peer review, potentially reducing the substantial delay of traditional review (avg. 13 weeks to first decision) without displacing human judgment.

Related papers

• PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
  PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
  arXiv  Detail & Related papers  (2026-09-15T11:08:11Z)
• AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality [39.33746637947531]
  We survey reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals.<n>We evaluate AI-generated peer reviews at ICLR 2026 and Nature Communications using a novel dataset.
  arXiv  Detail & Related papers  (2026-08-04T12:34:47Z)
• Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Community [33.39468805524881]
  We show that AI-mediated peer review is vulnerable to a simple, low-cost manipulation.<n>We see this across disciplines and publication venues, for both human-written and AI-generated papers.<n>Our results suggest that AI tools should not be treated as neutral evaluators in high-stakes peer review.
  arXiv  Detail & Related papers  (2026-06-08T20:38:06Z)
• Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews [11.21278193949403]
  We propose Sem-Detect, an authorship detection method for peer reviews.<n>It compares a target review against multiple AI-generated reviews of the same paper.<n>It is able to distinguish fully AI reviews from authentic human-written ones.
  arXiv  Detail & Related papers  (2026-05-20T20:19:16Z)
• On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists [113.03797263688519]
  Many scientists view AI reviewers as probabilistic systems without the expertise to evaluate research.<n>Existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts.
  arXiv  Detail & Related papers  (2026-05-20T03:33:55Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot [45.90272517740283]
  We report the first large-scale field deployment of AI-assisted peer review.<n>Every main-track submission at AAAI-26 received one clearly identified AI review from a state-of-the-art system.
  arXiv  Detail & Related papers  (2026-04-15T14:51:07Z)
• ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review [23.630458187587223]
  ReviewerToo is a framework for studying and deploying AI-assisted peer review.<n>It supports systematic experiments with specialized reviewer personas and structured evaluation criteria.<n>We show how AI can enhance consistency, coverage, and fairness while leaving complex evaluative judgments to domain experts.
  arXiv  Detail & Related papers  (2025-10-09T23:53:19Z)
• The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review [49.43514488610211]
  Author-provided rankings could be leveraged to improve peer review processes at machine learning conferences.<n>We focus on the Isotonic Mechanism, which calibrates raw review scores using the author-provided rankings.<n>We propose several cautious, low-risk applications of the Isotonic Mechanism and author-provided rankings in peer review.
  arXiv  Detail & Related papers  (2024-08-24T01:51:23Z)
• Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews [51.453135368388686]
  We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM) Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level.
  arXiv  Detail & Related papers  (2024-03-11T21:51:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.