Does double-blind peer-review reduce bias? Evidence from a top computer science conference

• URL: http://arxiv.org/abs/2101.02701v1
• Date: Thu, 7 Jan 2021 18:59:26 GMT
• Title: Does double-blind peer-review reduce bias? Evidence from a top computer science conference
• Authors: Mengyi Sun, Jainabou Barry Danfa, Misha Teplitskiy
• Abstract summary: We analyze the effects of double-blind peer review on prestige bias by analyzing the peer review files of 5027 papers submitted to the International Conference on Learning Representations. We find that after switching to double-blind review, the scores given to the most prestigious authors significantly decreased. We show that double-blind peer review may have improved the quality of the selections by limiting other (non-author-prestige) biases.
• Score: 2.642698101441705
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Peer review is widely regarded as essential for advancing scientific research. However, reviewers may be biased by authors' prestige or other characteristics. Double-blind peer review, in which the authors' identities are masked from the reviewers, has been proposed as a way to reduce reviewer bias. Although intuitive, evidence for the effectiveness of double-blind peer review in reducing bias is limited and mixed. Here, we examine the effects of double-blind peer review on prestige bias by analyzing the peer review files of 5027 papers submitted to the International Conference on Learning Representations (ICLR), a top computer science conference that changed its reviewing policy from single-blind peer review to double-blind peer review in 2018. We find that after switching to double-blind review, the scores given to the most prestigious authors significantly decreased. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance decisions significantly. Nevertheless, we show that double-blind peer review may have improved the quality of the selections by limiting other (non-author-prestige) biases. Specifically, papers rejected in the single-blind format are cited more than those rejected under the double-blind format, suggesting that double-blind review better identifies poorer quality papers. Interestingly, an apparently unrelated change - the change of rating scale from 10 to 4 points - likely reduced prestige bias significantly, to an extent that affected papers' acceptance. These results provide some support for the effectiveness of double-blind review in reducing prestige bias, while opening new research directions on the impact of peer review formats.

Related papers

• Semantic Properties of cosine based bias scores for word embeddings [52.13994416317707]
  We propose requirements for bias scores to be considered meaningful for quantifying biases. We analyze cosine based scores from the literature with regard to these requirements. We underline these findings with experiments to show that the bias scores' limitations have an impact in the application case.
  arXiv  Detail & Related papers  (2024-01-27T20:31:10Z)
• When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews [24.875901048855077]
  We introduce a novel task of automatically identifying contradictions among reviewers on a given article. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically.
  arXiv  Detail & Related papers  (2023-10-28T11:57:51Z)
• Estimating the Causal Effect of Early ArXiving on Paper Acceptance [56.538813945721685]
  We estimate the effect of arXiving a paper before the reviewing period (early arXiving) on its acceptance to the conference. Our results suggest that early arXiving may have a small effect on a paper's chances of acceptance.
  arXiv  Detail & Related papers  (2023-06-24T07:45:38Z)
• How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions? [87.00095008723181]
  Authors have roughly a three-fold overestimate of the acceptance probability of their papers. Female authors exhibit a marginally higher (statistically significant) miscalibration than male authors. At least 30% of respondents of both accepted and rejected papers said that their perception of their own paper improved after the review process.
  arXiv  Detail & Related papers  (2022-11-22T15:59:30Z)
• The SAME score: Improved cosine based bias score for word embeddings [63.24247894974291]
  We provide a bias definition based on the ideas from the literature and derive novel requirements for bias scores. We propose a new bias score, SAME, to address the shortcomings of existing bias scores and show empirically that SAME is better suited to quantify biases in word embeddings.
  arXiv  Detail & Related papers  (2022-03-28T09:28:13Z)
• Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment [26.30237757653724]
  We revisit the 2014 NeurIPS experiment that examined inconsistency in conference peer review. We find that for emphaccepted papers, there is no correlation between quality scores and impact of the paper.
  arXiv  Detail & Related papers  (2021-09-20T18:06:22Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)
• Can We Automate Scientific Reviewing? [89.50052670307434]
  We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
  arXiv  Detail & Related papers  (2021-01-30T07:16:53Z)
• Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review [35.24369486197371]
  Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review. Several conferences have started encouraging or even requiring authors to declare the previous submission history of their papers. We investigate whether reviewers exhibit a bias caused by the knowledge that the submission under review was previously rejected at a similar venue.
  arXiv  Detail & Related papers  (2020-11-30T09:35:37Z)
• Uncovering Latent Biases in Text: Method and Application to Peer Review [38.726731935235584]
  We introduce a novel framework to quantify bias in text caused by the visibility of subgroup membership indicators. We apply our framework to quantify biases in the text of peer reviews from a reputed machine learning conference.
  arXiv  Detail & Related papers  (2020-10-29T01:24:19Z)
• De-anonymization of authors through arXiv submissions during double-blind review [33.15282901539867]
  We investigate the effects of releasing arXiv preprints of papers undergoing a double-blind review process. We find statistically significant evidence of positive correlation between percentage acceptance and papers with high reputation released on arXiv.
  arXiv  Detail & Related papers  (2020-07-01T01:40:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.