NLPeer: A Unified Resource for the Computational Study of Peer Review

• URL: http://arxiv.org/abs/2211.06651v2
• Date: Fri, 19 May 2023 07:03:27 GMT
• Title: NLPeer: A Unified Resource for the Computational Study of Peer Review
• Authors: Nils Dycke, Ilia Kuznetsov, Iryna Gurevych
• Abstract summary: We introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues. We augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information. Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond.
• Score: 58.71736531356398
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Peer review constitutes a core component of scholarly publishing; yet it demands substantial expertise and training, and is susceptible to errors and biases. Various applications of NLP for peer reviewing assistance aim to support reviewers in this complex process, but the lack of clearly licensed datasets and multi-domain corpora prevent the systematic study of NLP for peer review. To remedy this, we introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues. In addition to the new datasets of paper drafts, camera-ready versions and peer reviews from the NLP community, we establish a unified data representation and augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information. We complement our resource with implementations and analysis of three reviewing assistance tasks, including a novel guided skimming task. Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond. The data and code are publicly available.

Related papers

• Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision [62.12545440385489]
  We introduce Re3, a framework for joint analysis of collaborative document revision. We present Re3-Sci, a large corpus of aligned scientific paper revisions manually labeled according to their action and intent. We use the new data to provide first empirical insights into collaborative document revision in the academic domain.
  arXiv  Detail & Related papers  (2024-05-31T21:19:09Z)
• What Can Natural Language Processing Do for Peer Review? [173.8912784451817]
  In modern science, peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review are largely text-based, Natural Language Processing has great potential to improve reviewing. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance.
  arXiv  Detail & Related papers  (2024-05-10T16:06:43Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [58.6354685593418]
  This paper proposes several article-level, field-normalized, and large language model-empowered bibliometric indicators to evaluate reviews. The newly emerging AI-generated literature reviews are also appraised. This work offers insights into the current challenges of literature reviews and envisions future directions for their development.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review [52.359007622096684]
  Peer review is a key component of the publishing process in most fields of science. Existing NLP studies focus on the analysis of individual texts. editorial assistance often requires modeling interactions between pairs of texts.
  arXiv  Detail & Related papers  (2022-04-22T16:39:38Z)
• NLPContributions: An Annotation Scheme for Machine Reading of Scholarly Contributions in Natural Language Processing Literature [0.0]
  We describe an annotation initiative to capture the scholarly contributions in natural language processing (NLP) articles. We develop the annotation task based on a pilot exercise on 50 NLP-ML scholarly articles presenting contributions to five information extraction tasks. We envision that the NLPContributions methodology engenders a wider discussion on the topic toward its further refinement and development.
  arXiv  Detail & Related papers  (2020-06-23T10:04:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.