Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions

• URL: http://arxiv.org/abs/2606.01013v2
• Date: Tue, 02 Jun 2026 06:07:26 GMT
• Title: Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions
• Abstract summary: We study 20 computer architecture papers, with varying levels of submission lineage, to expose how well AI review aligns with human review.<n>To conduct the case study, we build a web UI-integrated tool, emphAI-Paper-Review, that generates structured AI review of a draft paper.
• Score: 4.04252733799147
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Research is advancing faster than ever with artificial intelligence (AI); and so are the corresponding research papers. The exploding volume of AI-generated papers have put a strain to peer review, leading to the usage of AI-generated review, potentially wide yet sneaky. However, relevant ethical concerns about confidentiality, quality, and fairness are raised and no consensus has been reached in the broad research community. We expect the debate to continue for a while, but in the meantime, we ask an alternative, practical question: \textit{can AI review improve paper drafting?} We study 20 computer architecture papers, with varying levels of submission lineage, to expose how well AI review aligns with human review, quantified by a set of metrics we define. To conduct the case study, we build a web UI-integrated tool, \emph{AI-Paper-Review}, that generates structured AI review of a draft paper, available at https://github.com/unarylab/ai-paper-review. This tool selects several AI reviewers from a diverse pool of AI reviewers and clusters and ranks their comments based on commonality and importance of review comments. It also allows to align AI comments with human comments to facilitate metric-based validation. The case study shows that AI review can cover a significant fraction of human-raised issues, but also raises issues missing in human review. This paper is not intended to encourage using AI for peer review at the current stage, but to study that (1) how AI review can improve paper drafting and (2) the potential and limitation of AI-based peer review. The release of the tool and the case study data is intended to instigate future research on this topic. Misuse for peer review would violate the ethics policies from major academic venues.

Related papers

• Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Community [33.39468805524881]
  We show that AI-mediated peer review is vulnerable to a simple, low-cost manipulation.<n>We see this across disciplines and publication venues, for both human-written and AI-generated papers.<n>Our results suggest that AI tools should not be treated as neutral evaluators in high-stakes peer review.
  arXiv  Detail & Related papers  (2026-06-08T20:38:06Z)
• On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists [113.03797263688519]
  Many scientists view AI reviewers as probabilistic systems without the expertise to evaluate research.<n>Existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts.
  arXiv  Detail & Related papers  (2026-05-20T03:33:55Z)
• Stop Automating Peer Review Without Rigorous Evaluation [51.53099943385505]
  This position paper argues that today's AI systems should not be used to produce paper reviews.<n>We identify two critical issues: 1) AI reviewers exhibit a hivemind effect of excessive agreement within and across papers that reduces perspective diversity.<n>We argue that addressing the peer review crisis requires a science of peer review automation.
  arXiv  Detail & Related papers  (2026-05-04T22:41:04Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot [45.90272517740283]
  We report the first large-scale field deployment of AI-assisted peer review.<n>Every main-track submission at AAAI-26 received one clearly identified AI review from a state-of-the-art system.
  arXiv  Detail & Related papers  (2026-04-15T14:51:07Z)
• What happens when reviewers receive AI feedback in their reviews? [9.57486570505445]
  Advocates see AI's potential to reduce reviewer burden and improve quality, while critics warn of risks to fairness, accountability, and trust.<n>At ICLR 2025, an official AI feedback tool was deployed to provide reviewers with post-review suggestions.<n>This work contributes the first empirical evidence of such an AI tool in a live review process.
  arXiv  Detail & Related papers  (2026-02-14T15:22:33Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• The Great AI Witch Hunt: Reviewers Perception and (Mis)Conception of Generative AI in Research Writing [36.188062803005515]
  Generative AI (GenAI) use in research writing is growing fast.<n>It is unclear how peer reviewers recognize or misjudge AI-augmented manuscripts.<n>Our findings indicate that while AI-augmented writing improves readability, language diversity, and informativeness, it often lacks research details and reflective insights from authors.
  arXiv  Detail & Related papers  (2024-06-27T02:38:25Z)
• Can We Automate Scientific Reviewing? [89.50052670307434]
  We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
  arXiv  Detail & Related papers  (2021-01-30T07:16:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.