MOPRD: A multidisciplinary open peer review dataset
- URL: http://arxiv.org/abs/2212.04972v2
- Date: Tue, 14 Nov 2023 18:06:48 GMT
- Title: MOPRD: A multidisciplinary open peer review dataset
- Authors: Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, Xiaodong Shi
- Abstract summary: Open peer review is a growing trend in academic publications.
Most of the existing peer review datasets do not provide data that cover the whole peer review process.
We construct MOPRD, a multidisciplinary open peer review dataset.
- Score: 12.808751859133064
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Open peer review is a growing trend in academic publications. Public access
to peer review data can benefit both the academic and publishing communities.
It also serves as a great support to studies on review comment generation and
further to the realization of automated scholarly paper review. However, most
of the existing peer review datasets do not provide data that cover the whole
peer review process. Apart from this, their data are not diversified enough as
the data are mainly collected from the field of computer science. These two
drawbacks of the currently available peer review datasets need to be addressed
to unlock more opportunities for related studies. In response, we construct
MOPRD, a multidisciplinary open peer review dataset. This dataset consists of
paper metadata, multiple version manuscripts, review comments, meta-reviews,
author's rebuttal letters, and editorial decisions. Moreover, we propose a
modular guided review comment generation method based on MOPRD. Experiments
show that our method delivers better performance as indicated by both automatic
metrics and human evaluation. We also explore other potential applications of
MOPRD, including meta-review generation, editorial decision prediction, author
rebuttal generation, and scientometric analysis. MOPRD is a strong endorsement
for further studies in peer review-related research and other applications.
Related papers
- Deep Transfer Learning Based Peer Review Aggregation and Meta-review Generation for Scientific Articles [2.0778556166772986]
We address two peer review aggregation challenges: paper acceptance decision-making and meta-review generation.
Firstly, we propose to automate the process of acceptance decision prediction by applying traditional machine learning algorithms.
For the meta-review generation, we propose a transfer learning model based on the T5 model.
arXiv Detail & Related papers (2024-10-05T15:40:37Z) - AgentReview: Exploring Peer Review Dynamics with LLM Agents [13.826819101545926]
We introduce AgentReview, the first large language model (LLM) based peer review simulation framework.
Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases.
arXiv Detail & Related papers (2024-06-18T15:22:12Z) - Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions [62.0123588983514]
Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields.
We reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers.
We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources.
arXiv Detail & Related papers (2024-06-09T08:24:17Z) - A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [58.6354685593418]
This paper proposes several article-level, field-normalized, and large language model-empowered bibliometric indicators to evaluate reviews.
The newly emerging AI-generated literature reviews are also appraised.
This work offers insights into the current challenges of literature reviews and envisions future directions for their development.
arXiv Detail & Related papers (2024-02-20T11:28:50Z) - Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation [55.00687185394986]
We propose the task of scientific opinion summarization, where research paper reviews are synthesized into meta-reviews.
We introduce the ORSUM dataset covering 15,062 paper meta-reviews and 57,536 paper reviews from 47 conferences.
Our experiments show that (1) human-written summaries do not always satisfy all necessary criteria such as depth of discussion, and identifying consensus and controversy for the specific domain, and (2) the combination of task decomposition and iterative self-refinement shows strong potential for enhancing the opinions.
arXiv Detail & Related papers (2023-05-24T02:33:35Z) - NLPeer: A Unified Resource for the Computational Study of Peer Review [58.71736531356398]
We introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues.
We augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information.
Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond.
arXiv Detail & Related papers (2022-11-12T12:29:38Z) - Yes-Yes-Yes: Donation-based Peer Reviewing Data Collection for ACL
Rolling Review and Beyond [58.71736531356398]
We present an in-depth discussion of peer reviewing data, outline the ethical and legal desiderata for peer reviewing data collection, and propose the first continuous, donation-based data collection workflow.
We report on the ongoing implementation of this workflow at the ACL Rolling Review and deliver the first insights obtained with the newly collected data.
arXiv Detail & Related papers (2022-01-27T11:02:43Z) - Can We Automate Scientific Reviewing? [89.50052670307434]
We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers.
We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews.
Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
arXiv Detail & Related papers (2021-01-30T07:16:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.