Auctions and Prediction Markets for Scientific Peer Review
- URL: http://arxiv.org/abs/2109.00923v1
- Date: Fri, 27 Aug 2021 23:47:15 GMT
- Title: Auctions and Prediction Markets for Scientific Peer Review
- Authors: Siddarth Srinivasan, Jamie Morgenstern
- Abstract summary: We present a two-stage mechanism which ties together the paper submission and review process, simultaneously incentivizing high-quality reviews and high-quality submissions.
For the first stage, authors participate in a VCG auction for review slots by submitting their papers along with a bid that represents their expected value for having their paper reviewed.
For the second stage, we propose a novel prediction market-style mechanism (H-DIPP) building on recent work in the information elicitation literature, which incentivizes participating reviewers to provide honest and effortful reviews.
- Score: 9.904093594054679
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Peer reviewed publications are considered the gold standard in certifying and
disseminating ideas that a research community considers valuable. However, we
identify two major drawbacks of the current system: (1) the overwhelming demand
for reviewers due to a large volume of submissions, and (2) the lack of
incentives for reviewers to participate and expend the necessary effort to
provide high-quality reviews. In this work, we adopt a mechanism-design
approach to propose improvements to the peer review process. We present a
two-stage mechanism which ties together the paper submission and review
process, simultaneously incentivizing high-quality reviews and high-quality
submissions. In the first stage, authors participate in a VCG auction for
review slots by submitting their papers along with a bid that represents their
expected value for having their paper reviewed. For the second stage, we
propose a novel prediction market-style mechanism (H-DIPP) building on recent
work in the information elicitation literature, which incentivizes
participating reviewers to provide honest and effortful reviews. The revenue
raised by the Stage I auction is used in Stage II to pay reviewers based on the
quality of their reviews.
Related papers
- DecentPeeR: A Self-Incentivised & Inclusive Decentralized Peer Review System [11.477670199123335]
We show that our system, DecentPeeR, incentivizes reviewers to behave according to the rules, i.e., it has a unique Nash equilibrium in which virtuous behavior is rewarded.
arXiv Detail & Related papers (2024-11-13T09:13:16Z) - Analysis of the ICML 2023 Ranking Data: Can Authors' Opinions of Their Own Papers Assist Peer Review in Machine Learning? [52.00419656272129]
We conducted an experiment during the 2023 International Conference on Machine Learning (ICML)
We received 1,342 rankings, each from a distinct author, pertaining to 2,592 submissions.
We focus on the Isotonic Mechanism, which calibrates raw review scores using author-provided rankings.
arXiv Detail & Related papers (2024-08-24T01:51:23Z) - Decentralized Peer Review in Open Science: A Mechanism Proposal [0.0]
We propose a community-owned and -governed system for peer review.
The system aims to increase quality and speed of peer review while lowering the chance and impact of erroneous judgements.
arXiv Detail & Related papers (2024-04-28T11:42:54Z) - A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [58.6354685593418]
This paper proposes several article-level, field-normalized, and large language model-empowered bibliometric indicators to evaluate reviews.
The newly emerging AI-generated literature reviews are also appraised.
This work offers insights into the current challenges of literature reviews and envisions future directions for their development.
arXiv Detail & Related papers (2024-02-20T11:28:50Z) - Eliciting Honest Information From Authors Using Sequential Review [13.424398627546788]
We propose a sequential review mechanism that can truthfully elicit the ranking information from authors.
The key idea is to review the papers of an author in a sequence based on the provided ranking and conditioning the review of the next paper on the review scores of the previous papers.
arXiv Detail & Related papers (2023-11-24T17:27:39Z) - Submission-Aware Reviewer Profiling for Reviewer Recommender System [26.382772998002523]
We propose an approach that learns from each abstract published by a potential reviewer the topics studied and the explicit context in which the reviewer studied the topics.
Our experiments show a significant, consistent improvement in precision when compared with the existing methods.
The new approach has been deployed successfully at top-tier conferences in the last two years.
arXiv Detail & Related papers (2022-11-08T12:18:02Z) - Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer
Assignment [89.38213318211731]
Despite the benefits of using bids, reliance on paper bidding can allow malicious reviewers to manipulate the paper assignment for unethical purposes.
Several different approaches to preventing this manipulation have been proposed and deployed.
In this paper, we enumerate certain desirable properties that algorithms for addressing bid manipulation should satisfy.
arXiv Detail & Related papers (2022-07-22T19:58:17Z) - A Dataset on Malicious Paper Bidding in Peer Review [84.68308372858755]
Malicious reviewers strategically bid in order to unethically manipulate the paper assignment.
A critical impediment towards creating and evaluating methods to mitigate this issue is the lack of publicly-available data on malicious paper bidding.
We release a novel dataset, collected from a mock conference activity where participants were instructed to bid either honestly or maliciously.
arXiv Detail & Related papers (2022-06-24T20:23:33Z) - Ranking Scientific Papers Using Preference Learning [48.78161994501516]
We cast it as a paper ranking problem based on peer review texts and reviewer scores.
We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
arXiv Detail & Related papers (2021-09-02T19:41:47Z) - Making Paper Reviewing Robust to Bid Manipulation Attacks [44.34601846490532]
Anecdotal evidence suggests that some reviewers bid on papers by "friends" or colluding authors.
We develop a novel approach for paper bidding and assignment that is much more robust against such attacks.
In addition to being more robust, the quality of our paper review assignments is comparable to that of current, non-robust assignment approaches.
arXiv Detail & Related papers (2021-02-09T21:24:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.