Yes-Yes-Yes: Donation-based Peer Reviewing Data Collection for ACL
Rolling Review and Beyond
- URL: http://arxiv.org/abs/2201.11443v1
- Date: Thu, 27 Jan 2022 11:02:43 GMT
- Title: Yes-Yes-Yes: Donation-based Peer Reviewing Data Collection for ACL
Rolling Review and Beyond
- Authors: Nils Dycke, Ilia Kuznetsov, Iryna Gurevych
- Abstract summary: We present an in-depth discussion of peer reviewing data, outline the ethical and legal desiderata for peer reviewing data collection, and propose the first continuous, donation-based data collection workflow.
We report on the ongoing implementation of this workflow at the ACL Rolling Review and deliver the first insights obtained with the newly collected data.
- Score: 58.71736531356398
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Peer review is the primary gatekeeper of scientific merit and quality, yet it
is prone to bias and suffers from low efficiency. This demands
cross-disciplinary scrutiny of the processes that underlie peer reviewing;
however, quantitative research is limited by the data availability, as most of
the peer reviewing data across research disciplines is never made public.
Existing data collection efforts focus on few scientific domains and do not
address a range of ethical, license- and confidentiality-related issues
associated with peer reviewing data, preventing wide-scale research and
application development. While recent methods for peer review analysis and
processing show promise, a solid data foundation for computational research in
peer review is still missing. To address this, we present an in-depth
discussion of peer reviewing data, outline the ethical and legal desiderata for
peer reviewing data collection, and propose the first continuous,
donation-based data collection workflow that meets these requirements. We
report on the ongoing implementation of this workflow at the ACL Rolling Review
and deliver the first insights obtained with the newly collected data.
Related papers
- AgentReview: Exploring Peer Review Dynamics with LLM Agents [13.826819101545926]
We introduce AgentReview, the first large language model (LLM) based peer review simulation framework.
Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases.
arXiv Detail & Related papers (2024-06-18T15:22:12Z) - What Can Natural Language Processing Do for Peer Review? [173.8912784451817]
In modern science, peer review is widely used, yet it is hard, time-consuming, and prone to error.
Since the artifacts involved in peer review are largely text-based, Natural Language Processing has great potential to improve reviewing.
We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance.
arXiv Detail & Related papers (2024-05-10T16:06:43Z) - Automatic Analysis of Substantiation in Scientific Peer Reviews [24.422667012858298]
SubstanReview consists of 550 reviews from NLP conferences annotated by domain experts.
On the basis of this dataset, we train an argument mining system to automatically analyze the level of substantiation in peer reviews.
We also perform data analysis on the SubstanReview dataset to obtain meaningful insights on peer reviewing quality in NLP conferences over recent years.
arXiv Detail & Related papers (2023-11-20T17:47:37Z) - Assessing Scientific Contributions in Data Sharing Spaces [64.16762375635842]
This paper introduces the SCIENCE-index, a blockchain-based metric measuring a researcher's scientific contributions.
To incentivize researchers to share their data, the SCIENCE-index is augmented to include a data-sharing parameter.
Our model is evaluated by comparing the distribution of its output for geographically diverse researchers to that of the h-index.
arXiv Detail & Related papers (2023-03-18T19:17:47Z) - MOPRD: A multidisciplinary open peer review dataset [12.808751859133064]
Open peer review is a growing trend in academic publications.
Most of the existing peer review datasets do not provide data that cover the whole peer review process.
We construct MOPRD, a multidisciplinary open peer review dataset.
arXiv Detail & Related papers (2022-12-09T16:35:14Z) - NLPeer: A Unified Resource for the Computational Study of Peer Review [58.71736531356398]
We introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues.
We augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information.
Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond.
arXiv Detail & Related papers (2022-11-12T12:29:38Z) - Tag-Aware Document Representation for Research Paper Recommendation [68.8204255655161]
We propose a hybrid approach that leverages deep semantic representation of research papers based on social tags assigned by users.
The proposed model is effective in recommending research papers even when the rating data is very sparse.
arXiv Detail & Related papers (2022-09-08T09:13:07Z) - Ranking Scientific Papers Using Preference Learning [48.78161994501516]
We cast it as a paper ranking problem based on peer review texts and reviewer scores.
We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
arXiv Detail & Related papers (2021-09-02T19:41:47Z) - On the Privacy-Utility Tradeoff in Peer-Review Data Analysis [34.0435377376779]
A major impediment to research on improving peer review is the unavailability of peer-review data.
We propose a framework for privacy-preserving release of certain conference peer-review data.
arXiv Detail & Related papers (2020-06-29T21:08:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.