A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions

• URL: http://arxiv.org/abs/2011.15083v1
• Date: Mon, 30 Nov 2020 18:23:07 GMT
• Title: A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions
• Authors: Ivan Stelmakh, Charvi Rastogi, Nihar B. Shah, Aarti Singh, and Hal Daum\'e III
• Abstract summary: We aim to understand whether reviewers and more senior decision makers get disproportionately influenced by the first argument presented in a discussion. Specifically, we design and execute a randomized controlled trial with the goal of testing for the conditional causal effect of the discussion initiator's opinion on the outcome of a paper.
• Score: 33.261698377782075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing papers and making the final acceptance/rejection decisions. Given that human decision making is known to be susceptible to various cognitive biases, it is important to understand which (if any) biases are present in the peer-review process and design the pipeline such that the impact of these biases is minimized. In this work, we focus on the dynamics of between-reviewers discussions and investigate the presence of herding behaviour therein. In that, we aim to understand whether reviewers and more senior decision makers get disproportionately influenced by the first argument presented in the discussion when (in case of reviewers) they form an independent opinion about the paper before discussing it with others. Specifically, in conjunction with the review process of ICML 2020 -- a large, top tier machine learning conference -- we design and execute a randomized controlled trial with the goal of testing for the conditional causal effect of the discussion initiator's opinion on the outcome of a paper.

Related papers

• Estimating the Causal Effect of Early ArXiving on Paper Acceptance [56.538813945721685]
  We estimate the effect of arXiving a paper before the reviewing period (early arXiving) on its acceptance to the conference. Our results suggest that early arXiving may have a small effect on a paper's chances of acceptance.
  arXiv  Detail & Related papers  (2023-06-24T07:45:38Z)
• Perspectives on Large Language Models for Relevance Judgment [56.935731584323996]
  Large language models (LLMs) claim that they can assist with relevance judgments. It is not clear whether automated judgments can reliably be used in evaluations of retrieval systems.
  arXiv  Detail & Related papers  (2023-04-13T13:08:38Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)
• Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design [76.40919326501512]
  We consider the question: how should reviewers be divided between phases or conditions in order to maximize total assignment similarity? We empirically show that across several datasets pertaining to real conference data, dividing reviewers between phases/conditions uniformly at random allows an assignment that is nearly as good as the oracle optimal assignment.
  arXiv  Detail & Related papers  (2021-08-13T19:29:41Z)
• Argument Mining Driven Analysis of Peer-Reviews [4.552676857046446]
  We propose an Argument Mining based approach for the assistance of editors, meta-reviewers, and reviewers. One of our findings is that arguments used in the peer-review process differ from arguments in other domains making the transfer of pre-trained models difficult. We provide the community with a new peer-review dataset from different computer science conferences with annotated arguments.
  arXiv  Detail & Related papers  (2020-12-10T16:06:21Z)
• A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences [35.24369486197371]
  A surge in the number of submissions received by leading AI conferences has challenged the sustainability of the review process. We consider the problem of reviewer recruiting with a focus on the scarcity of qualified reviewers in large conferences. In conjunction with ICML 2020 -- a large, top-tier machine learning conference -- we recruit a small set of reviewers through our procedure and compare their performance with the general population of ICML reviewers.
  arXiv  Detail & Related papers  (2020-11-30T17:48:55Z)
• Analyzing the Machine Learning Conference Review Process [41.049292105761246]
  We critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020. Our findings suggest strong institutional bias in accept/reject decisions, even after controlling for paper quality. We find evidence for a gender gap, with female authors receiving lower scores, lower acceptance rates, and fewer citations per paper than their male counterparts.
  arXiv  Detail & Related papers  (2020-11-24T15:40:27Z)
• An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process [41.049292105761246]
  We critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020. Our findings suggest strong institutional bias in accept/reject decisions, even after controlling for paper quality. We find evidence for a gender gap, with female authors receiving lower scores, lower acceptance rates, and fewer citations per paper than their male counterparts.
  arXiv  Detail & Related papers  (2020-10-11T02:06:04Z)
• Aspect-based Sentiment Analysis of Scientific Reviews [12.472629584751509]
  We show that the distribution of aspect-based sentiments obtained from a review is significantly different for accepted and rejected papers. As a second objective, we quantify the extent of disagreement among the reviewers refereeing a paper. We also investigate the extent of disagreement between the reviewers and the chair and find that the inter-reviewer disagreement may have a link to the disagreement with the chair.
  arXiv  Detail & Related papers  (2020-06-05T07:06:01Z)
• Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals [53.484562601127195]
  We point out the inability to infer behavioral conclusions from probing results. We offer an alternative method that focuses on how the information is being used, rather than on what information is encoded.
  arXiv  Detail & Related papers  (2020-06-01T15:00:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.