Related papers: The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates

Related papers

Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Community [33.39468805524881]
We show that AI-mediated peer review is vulnerable to a simple, low-cost manipulation.<n>We see this across disciplines and publication venues, for both human-written and AI-generated papers.<n>Our results suggest that AI tools should not be treated as neutral evaluators in high-stakes peer review.
arXiv Detail & Related papers (2026-06-08T20:38:06Z)
Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions [4.04252733799147]
We study 20 computer architecture papers, with varying levels of submission lineage, to expose how well AI review aligns with human review.<n>To conduct the case study, we build a web UI-integrated tool, emphAI-Paper-Review, that generates structured AI review of a draft paper.
arXiv Detail & Related papers (2026-05-31T05:05:26Z)
On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists [113.03797263688519]
Many scientists view AI reviewers as probabilistic systems without the expertise to evaluate research.<n>Existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts.
arXiv Detail & Related papers (2026-05-20T03:33:55Z)
To Trust or Not to Trust: Authors' Response to AI-based Reviews [1.0435741631709405]
This paper reports findings from two independent pilot studies on authors' use and perceptions of AI-based auxiliary review.<n>Author support for future use was strongest when AI was framed as a supervised or author-controlled tool.
arXiv Detail & Related papers (2026-05-15T20:43:55Z)
Stop Automating Peer Review Without Rigorous Evaluation [51.53099943385505]
This position paper argues that today's AI systems should not be used to produce paper reviews.<n>We identify two critical issues: 1) AI reviewers exhibit a hivemind effect of excessive agreement within and across papers that reduces perspective diversity.<n>We argue that addressing the peer review crisis requires a science of peer review automation.
arXiv Detail & Related papers (2026-05-04T22:41:04Z)
When AI reviews science: Can we trust the referee? [73.47745294608072]
We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
arXiv Detail & Related papers (2026-04-26T08:03:32Z)
AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot [45.90272517740283]
We report the first large-scale field deployment of AI-assisted peer review.<n>Every main-track submission at AAAI-26 received one clearly identified AI review from a state-of-the-art system.
arXiv Detail & Related papers (2026-04-15T14:51:07Z)
What happens when reviewers receive AI feedback in their reviews? [9.57486570505445]
Advocates see AI's potential to reduce reviewer burden and improve quality, while critics warn of risks to fairness, accountability, and trust.<n>At ICLR 2025, an official AI feedback tool was deployed to provide reviewers with post-review suggestions.<n>This work contributes the first empirical evidence of such an AI tool in a live review process.
arXiv Detail & Related papers (2026-02-14T15:22:33Z)
ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review [23.630458187587223]
ReviewerToo is a framework for studying and deploying AI-assisted peer review.<n>It supports systematic experiments with specialized reviewer personas and structured evaluation criteria.<n>We show how AI can enhance consistency, coverage, and fairness while leaving complex evaluative judgments to domain experts.
arXiv Detail & Related papers (2025-10-09T23:53:19Z)
Everyone prefers human writers, including AI [0.0]
We conducted experiments using Raymond Queneaus Exercises Style (1947) to measure attribution bias.<n>Humans showed +13.7 percentage point (pp) bias (Cohen's h = 0.28, 95% CI: 0.21-0.34), while AI models showed +34.3 percentage point bias (h = 0.70, 95% CI: 0.65-0.76), a 2.5-fold stronger effect.
arXiv Detail & Related papers (2025-10-09T21:33:30Z)
CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
arXiv Detail & Related papers (2025-08-28T06:03:11Z)
The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
arXiv Detail & Related papers (2025-06-09T18:37:14Z)
AI Debate Aids Assessment of Controversial Claims [86.47978525513236]
We study whether AI debate can guide biased judges toward the truth by having two AI systems debate opposing sides of controversial COVID-19 factuality claims.<n>In our human study, we find that debate-where two AI advisor systems present opposing evidence-based arguments-consistently improves judgment accuracy and confidence calibration.<n>In our AI judge study, we find that AI judges with human-like personas achieve even higher accuracy (78.5%) than human judges (70.1%) and default AI judges without personas (69.8%)
arXiv Detail & Related papers (2025-06-02T19:01:53Z)
From Intuition to Understanding: Using AI Peers to Overcome Physics Misconceptions [8.60890432697274]
We designed a helpful AI "Peer" to help students correct fundamental physics misconceptions related to Newtonian mechanic concepts. In a randomized controlled trial with 165 students, those who engaged in targeted dialogue with the AI Peer achieved post-test scores that were, on average, 10.5 percentage points higher.
arXiv Detail & Related papers (2025-04-01T04:09:13Z)
Raising the Stakes: Performance Pressure Improves AI-Assisted Decision Making [57.53469908423318]
We show the effects of performance pressure on AI advice reliance when laypeople complete a common AI-assisted task. We find that when the stakes are high, people use AI advice more appropriately than when stakes are lower, regardless of the presence of an AI explanation.
arXiv Detail & Related papers (2024-10-21T22:39:52Z)
Analysis of the ICML 2023 Ranking Data: Can Authors' Opinions of Their Own Papers Assist Peer Review in Machine Learning? [52.00419656272129]
We conducted an experiment during the 2023 International Conference on Machine Learning (ICML) We received 1,342 rankings, each from a distinct author, pertaining to 2,592 submissions. We focus on the Isotonic Mechanism, which calibrates raw review scores using author-provided rankings.
arXiv Detail & Related papers (2024-08-24T01:51:23Z)
The Great AI Witch Hunt: Reviewers Perception and (Mis)Conception of Generative AI in Research Writing [25.73744132026804]
Generative AI (GenAI) use in research writing is growing fast. It is unclear how peer reviewers recognize or misjudge AI-augmented manuscripts. Our findings indicate that while AI-augmented writing improves readability, language diversity, and informativeness, it often lacks research details and reflective insights from authors.
arXiv Detail & Related papers (2024-06-27T02:38:25Z)
Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews [51.453135368388686]
We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM) Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level.
arXiv Detail & Related papers (2024-03-11T21:51:39Z)
When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews [24.875901048855077]
We introduce a novel task of automatically identifying contradictions among reviewers on a given article. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically.
arXiv Detail & Related papers (2023-10-28T11:57:51Z)
Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Review [4.081120388114928]
In the field of cybersecurity, the practice of double-blind peer review is the de-facto standard. This paper touches on the holy grail of peer reviewing and aims to shed light on the performance of AI in reviewing for academic security conferences. We investigate the predictability of reviewing outcomes by comparing the results obtained from human reviewers and machine-learning models.
arXiv Detail & Related papers (2023-09-11T13:51:40Z)
Fairness in AI and Its Long-Term Implications on Society [68.8204255655161]
We take a closer look at AI fairness and analyze how lack of AI fairness can lead to deepening of biases over time. We discuss how biased models can lead to more negative real-world outcomes for certain groups. If the issues persist, they could be reinforced by interactions with other risks and have severe implications on society in the form of social unrest.
arXiv Detail & Related papers (2023-04-16T11:22:59Z)
Metaethical Perspectives on 'Benchmarking' AI Ethics [81.65697003067841]
Benchmarks are seen as the cornerstone for measuring technical progress in Artificial Intelligence (AI) research. An increasingly prominent research area in AI is ethics, which currently has no set of benchmarks nor commonly accepted way for measuring the 'ethicality' of an AI system. We argue that it makes more sense to talk about 'values' rather than 'ethics' when considering the possible actions of present and future AI systems.
arXiv Detail & Related papers (2022-04-11T14:36:39Z)
Integrating Rankings into Quantized Scores in Peer Review [61.27794774537103]
In peer review, reviewers are usually asked to provide scores for the papers. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are no standard procedure for using this ranking information and Area Chairs may use it in different ways. We take a principled approach to integrate the ranking information into the scores.
arXiv Detail & Related papers (2022-04-05T19:39:13Z)
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review [35.24369486197371]
Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review. Several conferences have started encouraging or even requiring authors to declare the previous submission history of their papers. We investigate whether reviewers exhibit a bias caused by the knowledge that the submission under review was previously rejected at a similar venue.
arXiv Detail & Related papers (2020-11-30T09:35:37Z)
Evaluation Toolkit For Robustness Testing Of Automatic Essay Scoring Systems [64.4896118325552]
We evaluate the current state-of-the-art AES models using a model adversarial evaluation scheme and associated metrics. We find that AES models are highly overstable. Even heavy modifications(as much as 25%) with content unrelated to the topic of the questions do not decrease the score produced by the models.
arXiv Detail & Related papers (2020-07-14T03:49:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.