What Drives Paper Acceptance? A Process-Centric Analysis of Modern Peer Review

• URL: http://arxiv.org/abs/2509.25701v1
• Date: Tue, 30 Sep 2025 03:00:10 GMT
• Title: What Drives Paper Acceptance? A Process-Centric Analysis of Modern Peer Review
• Authors: Sangkeun Jung, Goun Pyeon, Inbum Heo, Hyungjin Ahn,
• Abstract summary: We present a large-scale empirical study of ICLR 2017-2025, encompassing over 28,000 submissions.<n>Our results show that factors beyond scientific novelty significantly shape acceptance outcomes.<n>We propose data-driven guidelines for authors, reviewers, and meta-reviewers to enhance transparency and fairness in peer review.
• Score: 2.9282248958475345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Peer review is the primary mechanism for evaluating scientific contributions, yet prior studies have mostly examined paper features or external metadata in isolation. The emergence of open platforms such as OpenReview has transformed peer review into a transparent and interactive process, recording not only scores and comments but also rebuttals, reviewer-author exchanges, reviewer disagreements, and meta-reviewer decisions. This provides unprecedented process-level data for understanding how modern peer review operates. In this paper, we present a large-scale empirical study of ICLR 2017-2025, encompassing over 28,000 submissions. Our analysis integrates four complementary dimensions, including the structure and language quality of papers (e.g., section patterns, figure/table ratios, clarity), submission strategies and external metadata (e.g., timing, arXiv posting, author count), the dynamics of author-reviewer interactions (e.g., rebuttal frequency, responsiveness), and the patterns of reviewer disagreement and meta-review mediation (e.g., score variance, confidence weighting). Our results show that factors beyond scientific novelty significantly shape acceptance outcomes. In particular, the rebuttal stage emerges as a decisive phase: timely, substantive, and interactive author-reviewer communication strongly increases the likelihood of acceptance, often outweighing initial reviewer skepticism. Alongside this, clearer writing, balanced visual presentation, earlier submission, and effective resolution of reviewer disagreement also correlate with higher acceptance probabilities. Based on these findings, we propose data-driven guidelines for authors, reviewers, and meta-reviewers to enhance transparency and fairness in peer review. Our study demonstrates that process-centric signals are essential for understanding and improving modern peer review.

Related papers

• EchoReview: Learning Peer Review from the Echoes of Scientific Citations [48.852960317704486]
  EchoReview is a citation-context-driven data synthesis framework.<n>It transforms scientific community's long-term judgments into structured review-style data.<n>It can achieve significant and stable improvements on core review dimensions such as evidence support and review comprehensiveness.
  arXiv  Detail & Related papers  (2026-01-31T13:55:38Z)
• Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time [55.756345497678204]
  We introduce a new framework for evidence-based comparative study of review quality.<n>We apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL.<n>We study the relationships between measurements of review quality, and its evolution over time.
  arXiv  Detail & Related papers  (2026-01-21T16:48:29Z)
• Insights from the ICLR Peer Review and Rebuttal Process [11.96840383745149]
  We present a large-scale analysis of the ICLR 2024 and 2025 peer review processes.<n>We examine review scores, author-reviewer engagement, temporal patterns in review submissions, and co-reviewer influence effects.<n>Our findings show that initial scores and the ratings of co-reviewers are the strongest predictors of score changes during the rebuttal.
  arXiv  Detail & Related papers  (2025-11-19T14:21:52Z)
• Identifying Aspects in Peer Reviews [61.374437855024844]
  We develop a data-driven schema for deriving aspects from a corpus of peer reviews.<n>We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)
• Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review [4.35783648216893]
  Traditional closed peer review systems are slow, costly, non-transparent, and possibly subject to biases.<n>We propose and examine the efficacy and accuracy of an alternative form of scientific peer review: through an open, bottom-up process.
  arXiv  Detail & Related papers  (2025-01-22T17:00:27Z)
• Deep Transfer Learning Based Peer Review Aggregation and Meta-review Generation for Scientific Articles [2.0778556166772986]
  We address two peer review aggregation challenges: paper acceptance decision-making and meta-review generation. Firstly, we propose to automate the process of acceptance decision prediction by applying traditional machine learning algorithms. For the meta-review generation, we propose a transfer learning model based on the T5 model.
  arXiv  Detail & Related papers  (2024-10-05T15:40:37Z)
• AgentReview: Exploring Peer Review Dynamics with LLM Agents [13.826819101545926]
  We introduce AgentReview, the first large language model (LLM) based peer review simulation framework. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases.
  arXiv  Detail & Related papers  (2024-06-18T15:22:12Z)
• GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews [25.291384842659397]
  We introduce sys, a summarization method designed to offer a concise yet comprehensive overview of scholarly reviews. Unlike traditional consensus-based methods, sys extracts both common and unique opinions from the reviews.
  arXiv  Detail & Related papers  (2024-06-11T15:27:01Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews [24.875901048855077]
  We introduce a novel task of automatically identifying contradictions among reviewers on a given article. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically.
  arXiv  Detail & Related papers  (2023-10-28T11:57:51Z)
• Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation [55.00687185394986]
  We propose the task of scientific opinion summarization, where research paper reviews are synthesized into meta-reviews. We introduce the ORSUM dataset covering 15,062 paper meta-reviews and 57,536 paper reviews from 47 conferences. Our experiments show that (1) human-written summaries do not always satisfy all necessary criteria such as depth of discussion, and identifying consensus and controversy for the specific domain, and (2) the combination of task decomposition and iterative self-refinement shows strong potential for enhancing the opinions.
  arXiv  Detail & Related papers  (2023-05-24T02:33:35Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)
• Aspect-based Sentiment Analysis of Scientific Reviews [12.472629584751509]
  We show that the distribution of aspect-based sentiments obtained from a review is significantly different for accepted and rejected papers. As a second objective, we quantify the extent of disagreement among the reviewers refereeing a paper. We also investigate the extent of disagreement between the reviewers and the chair and find that the inter-reviewer disagreement may have a link to the disagreement with the chair.
  arXiv  Detail & Related papers  (2020-06-05T07:06:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.