Automated scholarly paper review: Concepts, technologies, and challenges
- URL: http://arxiv.org/abs/2111.07533v4
- Date: Mon, 17 Jul 2023 17:16:42 GMT
- Title: Automated scholarly paper review: Concepts, technologies, and challenges
- Authors: Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, Xiaodong Shi
- Abstract summary: Recent years have seen the application of artificial intelligence (AI) in assisting the peer review process.
With the involvement of humans, such limitations remain inevitable.
- Score: 5.431798850623952
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Peer review is a widely accepted mechanism for research evaluation, playing a
pivotal role in academic publishing. However, criticisms have long been leveled
at this mechanism, mostly because of its poor efficiency and low
reproducibility. Recent years have seen the application of artificial
intelligence (AI) in assisting the peer review process. Nonetheless, with the
involvement of humans, such limitations remain inevitable. In this paper, we
propose the concept and pipeline of automated scholarly paper review (ASPR) and
review the relevant literature and technologies of achieving a full-scale
computerized review process. On the basis of the review and discussion, we
conclude that there is already corresponding research and preliminary
implementation at each stage of ASPR. We further look into the challenges in
ASPR with the existing technologies. The major difficulties lie in inadequate
data, imperfect document parsing and representation, defective
human$\unicode{x2013}$computer interaction, and flawed deep logical reasoning.
Moreover, we point out the future directions and discuss the possible moral and
ethical issues of ASPR. In the foreseeable future, ASPR and peer review will
coexist in a reinforcing manner before ASPR is able to fully undertake the
reviewing workload from humans.
Related papers
- What Can Natural Language Processing Do for Peer Review? [173.8912784451817]
In modern science, peer review is widely used, yet it is hard, time-consuming, and prone to error.
Since the artifacts involved in peer review are largely text-based, Natural Language Processing has great potential to improve reviewing.
We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance.
arXiv Detail & Related papers (2024-05-10T16:06:43Z) - A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [58.6354685593418]
This paper proposes several article-level, field-normalized, and large language model-empowered bibliometric indicators to evaluate reviews.
The newly emerging AI-generated literature reviews are also appraised.
This work offers insights into the current challenges of literature reviews and envisions future directions for their development.
arXiv Detail & Related papers (2024-02-20T11:28:50Z) - Responsible AI Considerations in Text Summarization Research: A Review
of Current Practices [89.85174013619883]
We focus on text summarization, a common NLP task largely overlooked by the responsible AI community.
We conduct a multi-round qualitative analysis of 333 summarization papers from the ACL Anthology published between 2020-2022.
We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals.
arXiv Detail & Related papers (2023-11-18T15:35:36Z) - A Critical Examination of the Ethics of AI-Mediated Peer Review [0.0]
Recent advancements in artificial intelligence (AI) systems offer promise and peril for scholarly peer review.
Human peer review systems are also fraught with related problems, such as biases, abuses, and a lack of transparency.
The legitimacy of AI-driven peer review hinges on the alignment with the scientific ethos.
arXiv Detail & Related papers (2023-09-02T18:14:10Z) - The Role of AI in Drug Discovery: Challenges, Opportunities, and
Strategies [97.5153823429076]
The benefits, challenges and drawbacks of AI in this field are reviewed.
The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
arXiv Detail & Related papers (2022-12-08T23:23:39Z) - Investigating Fairness Disparities in Peer Review: A Language Model
Enhanced Approach [77.61131357420201]
We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs)
We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date.
We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
arXiv Detail & Related papers (2022-11-07T16:19:42Z) - Generating Summaries for Scientific Paper Review [29.12631698162247]
The increase of submissions for top venues in machine learning and NLP has caused a problem of excessive burden on reviewers.
An automatic system for assisting with the reviewing process could be a solution for ameliorating the problem.
In this paper, we explore automatic review summary generation for scientific papers.
arXiv Detail & Related papers (2021-09-28T21:43:53Z) - LioNets: A Neural-Specific Local Interpretation Technique Exploiting
Penultimate Layer Information [6.570220157893279]
Interpretable machine learning (IML) is an urgent topic of research.
This paper focuses on a local-based, neural-specific interpretation process applied to textual and time-series data.
arXiv Detail & Related papers (2021-04-13T09:39:33Z) - Can We Automate Scientific Reviewing? [89.50052670307434]
We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers.
We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews.
Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
arXiv Detail & Related papers (2021-01-30T07:16:53Z) - State-of-Art-Reviewing: A Radical Proposal to Improve Scientific
Publication [19.10668029301668]
State-Of-the-Art Review (SOAR) is a neoteric reviewing pipeline that serves as a 'plug-and-play' replacement for peer review.
At the heart of our approach is an interpretation of the review process as a multi-objective, massively distributed and extremely-high-latency optimisation.
arXiv Detail & Related papers (2020-03-31T17:58:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.