Disparities in Peer Review Tone and the Role of Reviewer Anonymity

• URL: http://arxiv.org/abs/2507.14741v1
• Date: Sat, 19 Jul 2025 20:19:21 GMT
• Title: Disparities in Peer Review Tone and the Role of Reviewer Anonymity
• Authors: Maria Sahakyan, Bedoor AlShebli,
• Abstract summary: This study examines more than 80,000 reviews in two major journals.<n>It uncovers how review tone, sentiment, and supportive language vary across author demographics.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The peer review process is often regarded as the gatekeeper of scientific integrity, yet increasing evidence suggests that it is not immune to bias. Although structural inequities in peer review have been widely debated, much less attention has been paid to the subtle ways in which language itself may reinforce disparities. This study undertakes one of the most comprehensive linguistic analyses of peer review to date, examining more than 80,000 reviews in two major journals. Using natural language processing and large-scale statistical modeling, it uncovers how review tone, sentiment, and supportive language vary across author demographics, including gender, race, and institutional affiliation. Using a data set that includes both anonymous and signed reviews, this research also reveals how the disclosure of reviewer identity shapes the language of evaluation. The findings not only expose hidden biases in peer feedback, but also challenge conventional assumptions about anonymity's role in fairness. As academic publishing grapples with reform, these insights raise critical questions about how review policies shape career trajectories and scientific progress.

Related papers

• In-depth Research Impact Summarization through Fine-Grained Temporal Citation Analysis [52.42612945266194]
  We propose a new task: generating nuanced, expressive, and time-aware impact summaries.<n>We show that these summaries capture both praise (confirmation citations) and critique (correction citations) through the evolution of fine-grained citation intents.
  arXiv  Detail & Related papers  (2025-05-20T19:11:06Z)
• Identifying Aspects in Peer Reviews [61.374437855024844]
  We develop a data-driven schema for deriving aspects from a corpus of peer reviews.<n>We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)
• Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review [66.73247554182376]
  Large language models (LLMs) have led to their integration into peer review.<n>The unchecked adoption of LLMs poses significant risks to the integrity of the peer review system.<n>We show that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings.
  arXiv  Detail & Related papers  (2024-12-02T16:55:03Z)
• AgentReview: Exploring Peer Review Dynamics with LLM Agents [13.826819101545926]
  We introduce AgentReview, the first large language model (LLM) based peer review simulation framework. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases.
  arXiv  Detail & Related papers  (2024-06-18T15:22:12Z)
• GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews [25.291384842659397]
  We introduce sys, a summarization method designed to offer a concise yet comprehensive overview of scholarly reviews. Unlike traditional consensus-based methods, sys extracts both common and unique opinions from the reviews.
  arXiv  Detail & Related papers  (2024-06-11T15:27:01Z)
• What Can Natural Language Processing Do for Peer Review? [173.8912784451817]
  In modern science, peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review are largely text-based, Natural Language Processing has great potential to improve reviewing. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance.
  arXiv  Detail & Related papers  (2024-05-10T16:06:43Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [55.33653554387953]
  Pattern Analysis and Machine Intelligence (PAMI) has led to numerous literature reviews aimed at collecting and fragmented information.<n>This paper presents a thorough analysis of these literature reviews within the PAMI field.<n>We try to address three core research questions: (1) What are the prevalent structural and statistical characteristics of PAMI literature reviews; (2) What strategies can researchers employ to efficiently navigate the growing corpus of reviews; and (3) What are the advantages and limitations of AI-generated reviews compared to human-authored ones.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Polarity in the Classroom: A Case Study Leveraging Peer Sentiment Toward Scalable Assessment [4.588028371034406]
  Accurately grading open-ended assignments in large or massive open online courses (MOOCs) is non-trivial. In this work, we detail the process by which we create our domain-dependent lexicon and aspect-informed review form. We end by analyzing validity and discussing conclusions from our corpus of over 6800 peer reviews from nine courses.
  arXiv  Detail & Related papers  (2021-08-02T15:45:11Z)
• Uncovering Latent Biases in Text: Method and Application to Peer Review [38.726731935235584]
  We introduce a novel framework to quantify bias in text caused by the visibility of subgroup membership indicators. We apply our framework to quantify biases in the text of peer reviews from a reputed machine learning conference.
  arXiv  Detail & Related papers  (2020-10-29T01:24:19Z)
• Aspect-based Sentiment Analysis of Scientific Reviews [12.472629584751509]
  We show that the distribution of aspect-based sentiments obtained from a review is significantly different for accepted and rejected papers. As a second objective, we quantify the extent of disagreement among the reviewers refereeing a paper. We also investigate the extent of disagreement between the reviewers and the chair and find that the inter-reviewer disagreement may have a link to the disagreement with the chair.
  arXiv  Detail & Related papers  (2020-06-05T07:06:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.