PeeriScope: A Multi-Faceted Framework for Evaluating Peer Review Quality

• URL: http://arxiv.org/abs/2604.24071v1
• Date: Mon, 27 Apr 2026 05:56:56 GMT
• Title: PeeriScope: A Multi-Faceted Framework for Evaluating Peer Review Quality
• Abstract summary: We present PeeriScope, a modular platform that integrates structured features, rubric-guided large language model assessments, and supervised prediction to evaluate peer review quality along multiple dimensions.<n>The demonstration illustrates its use for reviewer self-assessment, editorial triage, and large-scale auditing, and it enables the continued development of quality evaluation methods within scientific peer review.
• Score: 12.173793674044383
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The increasing scale and variability of peer review in scholarly venues has created an urgent need for systematic, interpretable, and extensible tools to assess review quality. We present PeeriScope, a modular platform that integrates structured features, rubric-guided large language model assessments, and supervised prediction to evaluate peer review quality along multiple dimensions. Designed for openness and integration, PeeriScope provides both a public interface and a documented API, supporting practical deployment and research extensibility. The demonstration illustrates its use for reviewer self-assessment, editorial triage, and large-scale auditing, and it enables the continued development of quality evaluation methods within scientific peer review. PeeriScope is available both as a live demo at https://app.reviewer.ly/app/peeriscope and via API services at https://github.com/Reviewerly-Inc/Peeriscope.

Related papers

• Peerispect: Claim Verification in Scientific Peer Reviews [12.340116689292726]
  Peerispect is an interactive system that operationalizes claim-level verification in peer reviews.<n>Results are presented through a visual interface that highlights evidence directly in the paper.
  arXiv  Detail & Related papers  (2026-04-19T23:40:26Z)
• ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents [50.27474750319121]
  Review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews.<n>We introduce REVIEWBENCH, a benchmark evaluating review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews.<n>We propose REVIEWGROUNDER, a rubric-guided, tool-integrated multi-agent framework that decomposes reviewing into drafting and grounding stages.
  arXiv  Detail & Related papers  (2026-04-15T16:33:04Z)
• Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time [55.756345497678204]
  We introduce a new framework for evidence-based comparative study of review quality.<n>We apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL.<n>We study the relationships between measurements of review quality, and its evolution over time.
  arXiv  Detail & Related papers  (2026-01-21T16:48:29Z)
• Identifying Aspects in Peer Reviews [59.02879434536289]
  We develop a data-driven schema for deriving aspects from a corpus of peer reviews.<n>We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)
• Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions [62.0123588983514]
  Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields. We reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers. We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources.
  arXiv  Detail & Related papers  (2024-06-09T08:24:17Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements [167.73134600289603]
  evaluate is a library to support best practices for measurements, metrics, and comparisons of data and models. Evaluation on the Hub is a platform that enables the large-scale evaluation of over 75,000 models and 11,000 datasets.
  arXiv  Detail & Related papers  (2022-09-30T18:35:39Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)
• Systematic Review of Approaches to Improve Peer Assessment at Scale [5.067828201066184]
  This review focuses on three facets of Peer Assessment (PA) namely Auto grading and Peer Assessment Tools (we shall look only on how peer reviews/auto-grading is carried), strategies to handle Rogue Reviews, Peer Review Improvement using Natural Language Processing.
  arXiv  Detail & Related papers  (2020-01-27T15:59:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.