Traditional statistical representations outperform generative AI in identifying expert peer reviewers

• URL: http://arxiv.org/abs/2605.18752v1
• Date: Mon, 18 May 2026 17:59:45 GMT
• Title: Traditional statistical representations outperform generative AI in identifying expert peer reviewers
• Abstract summary: The exponential growth of scientific submissions has strained the peer review system.<n>We evaluate the reliability of statistical and AI-driven expertise identification methodologies.<n>Our results highlight that distinguishing subfield expertise requires fine-grained vocabulary.
• Score: 22.555352928759206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The exponential growth of scientific submissions has strained the peer review system. Despite the rapidly expanding global pool of researchers, this unprecedented scale has rendered the previous approach of manual expert identification unfeasible. Therefore, institutions have naturally turned to Large Language Models (LLMs) to automate intricate processes like expert reviewer identification. However, the reliability of these new models in accurately identifying domain experts lacks rigorous evaluation. We conduct a comprehensive empirical evaluation of statistical and AI-driven expertise identification methodologies to benchmark their reliability and limitations. Framing expert identification as an information retrieval problem, we utilize the distributed peer review system of a major international astronomical observatory, where proposal authorship serves as our proxy ground truth for domain expertise. Evaluating six retrieval methodologies utilized across observatories and computer science conferences, we demonstrate that traditional statistical representations outperform generative AI. Specifically, Term Frequency-Inverse Document Frequency successfully identified a labeled expert within the top 25 recommendations 79.5% of the time, compared to 51.5% for GPT-4o mini. Our results highlight that distinguishing subfield expertise requires fine-grained vocabulary, which is obscured by the semantic smoothing in generative methods. By establishing a rigorous evaluation framework for automated peer review, we demonstrate that transparent and reproducible statistical representations still outperform computationally expensive LLMs in specialized scientific tasks.

Related papers

• LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges [4.655159257282136]
  This survey offers a systems-level analysis of large language models (LLMs)-based scientific peer review.<n>We focus on two core evaluative functions: critique generation and score prediction.<n>We identify emerging risks, including prompt injection, data poisoning, retrieval vulnerabilities, and reward hacking.
  arXiv  Detail & Related papers  (2026-06-23T18:12:39Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation [32.5154721488471]
  Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks.<n>Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases.<n>We present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains.
  arXiv  Detail & Related papers  (2026-03-27T11:28:15Z)
• An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems [1.9138416746729587]
  Large Language Models (LLMs) are transforming scholarly tasks like search and summarization, but their reliability remains uncertain.<n>We developed a schema for evaluating LLM errors in question-answering systems that reflects the assessment strategies of practicing scientists.
  arXiv  Detail & Related papers  (2026-02-24T16:16:44Z)
• FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights [63.32178443510396]
  We introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings.<n>Even the strongest agents achieve limited rediscovery success (50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-02-02T23:21:13Z)
• How to Build AI Agents by Augmenting LLMs with Codified Human Expert Domain Knowledge? A Software Engineering Framework [3.0049184484925604]
  Critical domain knowledge typically resides with few experts.<n>Non-experts struggle to create effective visualizations.<n>This paper investigates how to capture and embed human domain knowledge into AI agent systems.
  arXiv  Detail & Related papers  (2026-01-21T16:23:22Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research [81.04845910798387]
  Generating natural language explanations for threat detections remains an open problem in cybersecurity research.<n>We present AutoMalDesc, an automated static analysis summarization framework that operates independently at scale.<n>We publish our complete dataset of more than 100K script samples, including annotated seed (0.9K) datasets, along with our methodology and evaluation framework.
  arXiv  Detail & Related papers  (2025-11-17T13:05:25Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• On the Biased Assessment of Expert Finding Systems [11.083396379885478]
  In large organisations, identifying experts on a given topic is crucial in leveraging the internal knowledge spread across teams and departments. This case study provides an analysis of how these recommendations can impact the evaluation of expert finding systems. We show that system-validated annotations lead to overestimated performance of traditional term-based retrieval models. We also augment knowledge areas with synonyms to uncover a strong bias towards literal mentions of their constituent words.
  arXiv  Detail & Related papers  (2024-10-07T13:19:08Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Firenze: Model Evaluation Using Weak Signals [5.723905680436377]
  We introduce Firenze, a novel framework for comparative evaluation of machine learning models' performance. We show that markers computed and combined over select subsets of samples called regions of interest can provide a robust estimate of their real-world performances.
  arXiv  Detail & Related papers  (2022-07-02T13:20:38Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.