Automatic Analysis of Substantiation in Scientific Peer Reviews
- URL: http://arxiv.org/abs/2311.11967v1
- Date: Mon, 20 Nov 2023 17:47:37 GMT
- Title: Automatic Analysis of Substantiation in Scientific Peer Reviews
- Authors: Yanzhu Guo, Guokan Shang, Virgile Rennard, Michalis Vazirgiannis and
Chlo\'e Clavel
- Abstract summary: SubstanReview consists of 550 reviews from NLP conferences annotated by domain experts.
On the basis of this dataset, we train an argument mining system to automatically analyze the level of substantiation in peer reviews.
We also perform data analysis on the SubstanReview dataset to obtain meaningful insights on peer reviewing quality in NLP conferences over recent years.
- Score: 24.422667012858298
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: With the increasing amount of problematic peer reviews in top AI conferences,
the community is urgently in need of automatic quality control measures. In
this paper, we restrict our attention to substantiation -- one popular quality
aspect indicating whether the claims in a review are sufficiently supported by
evidence -- and provide a solution automatizing this evaluation process. To
achieve this goal, we first formulate the problem as claim-evidence pair
extraction in scientific peer reviews, and collect SubstanReview, the first
annotated dataset for this task. SubstanReview consists of 550 reviews from NLP
conferences annotated by domain experts. On the basis of this dataset, we train
an argument mining system to automatically analyze the level of substantiation
in peer reviews. We also perform data analysis on the SubstanReview dataset to
obtain meaningful insights on peer reviewing quality in NLP conferences over
recent years.
Related papers
- Analysis of the ICML 2023 Ranking Data: Can Authors' Opinions of Their Own Papers Assist Peer Review in Machine Learning? [52.00419656272129]
We conducted an experiment during the 2023 International Conference on Machine Learning (ICML)
We received 1,342 rankings, each from a distinct author, pertaining to 2,592 submissions.
We focus on the Isotonic Mechanism, which calibrates raw review scores using author-provided rankings.
arXiv Detail & Related papers (2024-08-24T01:51:23Z) - What Can Natural Language Processing Do for Peer Review? [173.8912784451817]
In modern science, peer review is widely used, yet it is hard, time-consuming, and prone to error.
Since the artifacts involved in peer review are largely text-based, Natural Language Processing has great potential to improve reviewing.
We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance.
arXiv Detail & Related papers (2024-05-10T16:06:43Z) - When Reviewers Lock Horn: Finding Disagreement in Scientific Peer
Reviews [24.875901048855077]
We introduce a novel task of automatically identifying contradictions among reviewers on a given article.
To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically.
arXiv Detail & Related papers (2023-10-28T11:57:51Z) - NLPeer: A Unified Resource for the Computational Study of Peer Review [58.71736531356398]
We introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues.
We augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information.
Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond.
arXiv Detail & Related papers (2022-11-12T12:29:38Z) - Submission-Aware Reviewer Profiling for Reviewer Recommender System [26.382772998002523]
We propose an approach that learns from each abstract published by a potential reviewer the topics studied and the explicit context in which the reviewer studied the topics.
Our experiments show a significant, consistent improvement in precision when compared with the existing methods.
The new approach has been deployed successfully at top-tier conferences in the last two years.
arXiv Detail & Related papers (2022-11-08T12:18:02Z) - Investigating Fairness Disparities in Peer Review: A Language Model
Enhanced Approach [77.61131357420201]
We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs)
We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date.
We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
arXiv Detail & Related papers (2022-11-07T16:19:42Z) - Yes-Yes-Yes: Donation-based Peer Reviewing Data Collection for ACL
Rolling Review and Beyond [58.71736531356398]
We present an in-depth discussion of peer reviewing data, outline the ethical and legal desiderata for peer reviewing data collection, and propose the first continuous, donation-based data collection workflow.
We report on the ongoing implementation of this workflow at the ACL Rolling Review and deliver the first insights obtained with the newly collected data.
arXiv Detail & Related papers (2022-01-27T11:02:43Z) - Generating Summaries for Scientific Paper Review [29.12631698162247]
The increase of submissions for top venues in machine learning and NLP has caused a problem of excessive burden on reviewers.
An automatic system for assisting with the reviewing process could be a solution for ameliorating the problem.
In this paper, we explore automatic review summary generation for scientific papers.
arXiv Detail & Related papers (2021-09-28T21:43:53Z) - Ranking Scientific Papers Using Preference Learning [48.78161994501516]
We cast it as a paper ranking problem based on peer review texts and reviewer scores.
We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
arXiv Detail & Related papers (2021-09-02T19:41:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.