Related papers: Group Fairness in Peer Review

Group Fairness in Peer Review

URL: http://arxiv.org/abs/2410.03474v1
Date: Fri, 4 Oct 2024 14:48:10 GMT
Title: Group Fairness in Peer Review
Authors: Haris Aziz, Evi Micha, Nisarg Shah,
Abstract summary: This paper introduces a notion of group fairness, called the core, which requires that every possible community (subset of researchers) be treated in a way that prevents them from unilaterally benefiting from withdrawing from a large conference. We study a simple peer review model, prove that it always admits a reviewing assignment in the core, and design an efficient algorithm to find one such assignment.
Score: 44.580732477017904
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Large conferences such as NeurIPS and AAAI serve as crossroads of various AI fields, since they attract submissions from a vast number of communities. However, in some cases, this has resulted in a poor reviewing experience for some communities, whose submissions get assigned to less qualified reviewers outside of their communities. An often-advocated solution is to break up any such large conference into smaller conferences, but this can lead to isolation of communities and harm interdisciplinary research. We tackle this challenge by introducing a notion of group fairness, called the core, which requires that every possible community (subset of researchers) to be treated in a way that prevents them from unilaterally benefiting by withdrawing from a large conference. We study a simple peer review model, prove that it always admits a reviewing assignment in the core, and design an efficient algorithm to find one such assignment. We use real data from CVPR and ICLR conferences to compare our algorithm to existing reviewing assignment algorithms on a number of metrics.

Related papers

On the Detection of Reviewer-Author Collusion Rings From Paper Bidding [71.43634536456844]
Collusion rings pose a major threat to the peer-review systems of computer science conferences. One approach to solve this problem would be to detect the colluding reviewers from their manipulated bids. No research has yet established that detecting collusion rings is even possible.
arXiv Detail & Related papers (2024-02-12T18:12:09Z)
Robust Consensus Clustering and its Applications for Advertising Forecasting [18.242055675730253]
We propose a novel algorithm -- robust consensus clustering that can find common ground truth among experts' opinions. We apply the proposed method to the real-world advertising campaign segmentation and forecasting tasks.
arXiv Detail & Related papers (2022-12-27T21:49:04Z)
Spatio-Temporal Graph Representation Learning for Fraudster Group Detection [50.779498955162644]
Companies may hire fraudster groups to write fake reviews to either demote competitors or promote their own businesses. To detect such groups, a common model is to represent fraudster groups' static networks. We propose to first capitalize on the effectiveness of the HIN-RNN in both reviewers' representation learning.
arXiv Detail & Related papers (2022-01-07T08:01:38Z)
Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design [76.40919326501512]
We consider the question: how should reviewers be divided between phases or conditions in order to maximize total assignment similarity? We empirically show that across several datasets pertaining to real conference data, dividing reviewers between phases/conditions uniformly at random allows an assignment that is nearly as good as the oracle optimal assignment.
arXiv Detail & Related papers (2021-08-13T19:29:41Z)
Some Ethical Issues in the Review Process of Machine Learning Conferences [0.38073142980733]
Recent successes in the Machine Learning community have led to a steep increase in the number of papers submitted to conferences. This increase made more prominent some of the issues that affect the current review process used by these conferences. We study the problem of reviewers' recruitment, infringements of the double-blind process, fraudulent behaviors, biases in numerical ratings, and the appendix phenomenon.
arXiv Detail & Related papers (2021-06-01T21:22:41Z)
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences [35.24369486197371]
A surge in the number of submissions received by leading AI conferences has challenged the sustainability of the review process. We consider the problem of reviewer recruiting with a focus on the scarcity of qualified reviewers in large conferences. In conjunction with ICML 2020 -- a large, top-tier machine learning conference -- we recruit a small set of reviewers through our procedure and compare their performance with the general population of ICML reviewers.
arXiv Detail & Related papers (2020-11-30T17:48:55Z)
What Can We Do to Improve Peer Review in NLP? [69.11622020605431]
We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing apples-to-oranges comparisons. There are several potential ways forward, but the key difficulty is creating the incentives and mechanisms for their consistent implementation in the NLP community.
arXiv Detail & Related papers (2020-10-08T09:32:21Z)
Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments [96.114824979298]
Three important challenges in conference peer review are maliciously attempting to get assigned to certain papers and "torpedo reviewing" We present a framework that brings all these challenges under a common umbrella and present a (randomized) algorithm for reviewer assignment. Our algorithms can limit the chance that any malicious reviewer gets assigned to their desired paper to 50% while producing assignments with over 90% of the total optimal similarity.
arXiv Detail & Related papers (2020-06-29T23:55:53Z)
A Correspondence Analysis Framework for Author-Conference Recommendations [2.1055643409860743]
We use Correspondence Analysis (CA) to derive appropriate relationships between the entities in question, such as conferences and papers. Our models show promising results when compared with existing methods such as content-based filtering, collaborative filtering and hybrid filtering.
arXiv Detail & Related papers (2020-01-08T18:52:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.