OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews

• URL: http://arxiv.org/abs/2412.11948v3
• Date: Tue, 18 Mar 2025 08:37:47 GMT
• Title: OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews
• Authors: Maximilian Idahl, Zahra Ahmadi,
• Abstract summary: OpenReviewer is an open-source system for generating high-quality peer reviews of machine learning and AI conference papers.<n>Llama-OpenReviewer-8B is an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences.
• Score: 3.660182910533372
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review following conference-specific guidelines. Our evaluation on 400 test papers shows that OpenReviewer produces considerably more critical and realistic reviews compared to general-purpose LLMs like GPT-4 and Claude-3.5. While other LLMs tend toward overly positive assessments, OpenReviewer's recommendations closely match the distribution of human reviewer ratings. The system provides authors with rapid, constructive feedback to improve their manuscripts before submission, though it is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.

Related papers

• LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews [74.87393214734114]
  This work introduces LazyReview, a dataset of peer-review sentences annotated with fine-grained lazy thinking categories. Large Language Models (LLMs) struggle to detect these instances in a zero-shot setting. instruction-based fine-tuning on our dataset significantly boosts performance by 10-20 performance points.
  arXiv  Detail & Related papers  (2025-04-15T10:07:33Z)
• Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 [115.86204862475864]
  Review Feedback Agent provides automated feedback on vague comments, content misunderstandings, and unprofessional remarks to reviewers. It was implemented at ICLR 2025 as a large randomized control study. 27% of reviewers who received feedback updated their reviews, and over 12,000 feedback suggestions from the agent were incorporated by those reviewers.
  arXiv  Detail & Related papers  (2025-04-13T22:01:25Z)
• DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process [30.710131188931317]
  DeepReview is a framework designed to emulate expert reviewers by incorporating structured analysis, literature retrieval, and evidence-based argumentation. In its best mode, DeepReviewer-14B achieves win rates of 88.21% and 80.20% against GPT-o1 and DeepSeek-R1 in evaluations.
  arXiv  Detail & Related papers  (2025-03-11T15:59:43Z)
• Generative Adversarial Reviews: When LLMs Become the Critic [1.2430809884830318]
  We introduce Generative Agent Reviewers (GAR), leveraging LLM-empowered agents to simulate faithful peer reviewers.<n>Central to this approach is a graph-based representation of manuscripts, condensing content and logically organizing information.<n>Our experiments demonstrate that GAR performs comparably to human reviewers in providing detailed feedback and predicting paper outcomes.
  arXiv  Detail & Related papers  (2024-12-09T06:58:17Z)
• Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review [66.73247554182376]
  Large language models (LLMs) have led to their integration into peer review.<n>The unchecked adoption of LLMs poses significant risks to the integrity of the peer review system.<n>We show that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings.
  arXiv  Detail & Related papers  (2024-12-02T16:55:03Z)
• Streamlining the review process: AI-generated annotations in research manuscripts [0.5735035463793009]
  This study explores the potential of integrating Large Language Models (LLMs) into the peer-review process to enhance efficiency without compromising effectiveness.<n>We focus on manuscript annotations, particularly excerpt highlights, as a potential area for AI-human collaboration.<n>This paper introduces AnnotateGPT, a platform that utilizes GPT-4 for manuscript review, aiming to improve reviewers' comprehension and focus.
  arXiv  Detail & Related papers  (2024-11-29T23:26:34Z)
• AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews [18.50142644126276]
  We evaluate the alignment of automatic paper reviews with human reviews using an arena of human preferences by pairwise comparisons. We fine-tune an LLM to predict human preferences, predicting which reviews humans will prefer in a head-to-head battle between LLMs. We make the reviews of publicly available arXiv and open-access Nature journal papers available online, along with a free service which helps authors review and revise their research papers and improve their quality.
  arXiv  Detail & Related papers  (2024-08-19T19:10:38Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [55.33653554387953]
  Pattern Analysis and Machine Intelligence (PAMI) has led to numerous literature reviews aimed at collecting and fragmented information.<n>This paper presents a thorough analysis of these literature reviews within the PAMI field.<n>We try to address three core research questions: (1) What are the prevalent structural and statistical characteristics of PAMI literature reviews; (2) What strategies can researchers employ to efficiently navigate the growing corpus of reviews; and (3) What are the advantages and limitations of AI-generated reviews compared to human-authored ones.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Reviewer2: Optimizing Review Generation Through Prompt Generation [28.050468098801872]
  We propose an efficient two-stage review generation framework called Reviewer2.<n>Unlike prior work, this approach explicitly models the distribution of possible aspects that the review may address.<n>We generate a large-scale review dataset of 27k papers and 99k reviews that we annotate with aspect prompts.
  arXiv  Detail & Related papers  (2024-02-16T18:43:10Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)
• Can We Automate Scientific Reviewing? [89.50052670307434]
  We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
  arXiv  Detail & Related papers  (2021-01-30T07:16:53Z)
• Unsupervised Opinion Summarization with Noising and Denoising [85.49169453434554]
  We create a synthetic dataset from a corpus of user reviews by sampling a review, pretending it is a summary, and generating noisy versions thereof. At test time, the model accepts genuine reviews and generates a summary containing salient opinions, treating those that do not reach consensus as noise.
  arXiv  Detail & Related papers  (2020-04-21T16:54:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.