Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review

• URL: http://arxiv.org/abs/2011.14646v1
• Date: Mon, 30 Nov 2020 09:35:37 GMT
• Title: Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
• Authors: Ivan Stelmakh, Nihar B. Shah, Aarti Singh, and Hal Daum\'e III
• Abstract summary: Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review. Several conferences have started encouraging or even requiring authors to declare the previous submission history of their papers. We investigate whether reviewers exhibit a bias caused by the knowledge that the submission under review was previously rejected at a similar venue.
• Score: 35.24369486197371
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent reviewers is growing at a much slower rate. To curb this trend and reduce the burden on reviewers, several conferences have started encouraging or even requiring authors to declare the previous submission history of their papers. Such initiatives have been met with skepticism among authors, who raise the concern about a potential bias in reviewers' recommendations induced by this information. In this work, we investigate whether reviewers exhibit a bias caused by the knowledge that the submission under review was previously rejected at a similar venue, focusing on a population of novice reviewers who constitute a large fraction of the reviewer pool in leading machine learning and computer science conferences. We design and conduct a randomized controlled trial closely replicating the relevant components of the peer-review pipeline with $133$ reviewers (master's, junior PhD students, and recent graduates of top US universities) writing reviews for $19$ papers. The analysis reveals that reviewers indeed become negatively biased when they receive a signal about paper being a resubmission, giving almost 1 point lower overall score on a 10-point Likert item ($\Delta = -0.78, \ 95\% \ \text{CI} = [-1.30, -0.24]$) than reviewers who do not receive such a signal. Looking at specific criteria scores (originality, quality, clarity and significance), we observe that novice reviewers tend to underrate quality the most.

Related papers

• Analysis of the ICML 2023 Ranking Data: Can Authors' Opinions of Their Own Papers Assist Peer Review in Machine Learning? [52.00419656272129]
  We conducted an experiment during the 2023 International Conference on Machine Learning (ICML) We received 1,342 rankings, each from a distinct author, pertaining to 2,592 submissions. We focus on the Isotonic Mechanism, which calibrates raw review scores using author-provided rankings.
  arXiv  Detail & Related papers  (2024-08-24T01:51:23Z)
• When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews [24.875901048855077]
  We introduce a novel task of automatically identifying contradictions among reviewers on a given article. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically.
  arXiv  Detail & Related papers  (2023-10-28T11:57:51Z)
• How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions? [87.00095008723181]
  Authors have roughly a three-fold overestimate of the acceptance probability of their papers. Female authors exhibit a marginally higher (statistically significant) miscalibration than male authors. At least 30% of respondents of both accepted and rejected papers said that their perception of their own paper improved after the review process.
  arXiv  Detail & Related papers  (2022-11-22T15:59:30Z)
• Spatio-Temporal Graph Representation Learning for Fraudster Group Detection [50.779498955162644]
  Companies may hire fraudster groups to write fake reviews to either demote competitors or promote their own businesses. To detect such groups, a common model is to represent fraudster groups' static networks. We propose to first capitalize on the effectiveness of the HIN-RNN in both reviewers' representation learning.
  arXiv  Detail & Related papers  (2022-01-07T08:01:38Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)
• Can We Automate Scientific Reviewing? [89.50052670307434]
  We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
  arXiv  Detail & Related papers  (2021-01-30T07:16:53Z)
• Does double-blind peer-review reduce bias? Evidence from a top computer science conference [2.642698101441705]
  We analyze the effects of double-blind peer review on prestige bias by analyzing the peer review files of 5027 papers submitted to the International Conference on Learning Representations. We find that after switching to double-blind review, the scores given to the most prestigious authors significantly decreased. We show that double-blind peer review may have improved the quality of the selections by limiting other (non-author-prestige) biases.
  arXiv  Detail & Related papers  (2021-01-07T18:59:26Z)
• Debiasing Evaluations That are Biased by Evaluations [32.135315382120154]
  We consider the problem of mitigating outcome-induced biases in ratings when some information about the outcome is available. We propose a debiasing method by solving a regularized optimization problem under this ordering constraint. We also provide a carefully designed cross-validation method that adaptively chooses the appropriate amount of regularization.
  arXiv  Detail & Related papers  (2020-12-01T18:20:43Z)
• Understanding Peer Review of Software Engineering Papers [5.744593856232663]
  We aim at understanding how reviewers, including those who have won awards for reviewing, perform their reviews of software engineering papers. The most important features of papers that result in positive reviews are clear and supported validation, an interesting problem, and novelty. Authors should make the contribution of the work very clear in their paper.
  arXiv  Detail & Related papers  (2020-09-02T17:31:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.