HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews

• URL: http://arxiv.org/abs/2609.03580v1
• Date: Thu, 03 Sep 2026 09:27:54 GMT
• Title: HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews
• Abstract summary: HalluPeer is a benchmark for detecting hallucinations in scientific peer reviews.<n>Our pipeline induces a peer-review-specific hallucination taxonomy, identifies review contexts, and injects hallucinations with automated filtering.
• Score: 20.989630888021463
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The growing scale of academic peer review has motivated the use of Large Language Models (LLMs) as review assistants, yet LLMs can generate fluent but unsupported claims that undermine review reliability. Existing hallucination benchmarks are not designed for peer review, where verification requires grounding claims in long, technical papers. We introduce HalluPeer, a benchmark for detecting hallucinations in scientific peer reviews, providing aligned triples of paper content, human-written reviews, and hallucination-injected reviews, annotated for detection, classification, and localization. Our pipeline induces a peer-review-specific hallucination taxonomy, identifies review contexts, and injects hallucinations with automated filtering. Experiments on 12K papers and 38K reviews show that existing detectors struggle to separate hallucinations from legitimate critique, while evaluation on authentic reviews demonstrates that HalluPeer-defined hallucination patterns occur in real peer reviews, highlighting the critical need for source-aware verification. Our project page can be found in https://github.com/Lin-TzuLing/HalluPeer.git

Related papers

• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• Peerispect: Claim Verification in Scientific Peer Reviews [12.340116689292726]
  Peerispect is an interactive system that operationalizes claim-level verification in peer reviews.<n>Results are presented through a visual interface that highlights evidence directly in the paper.
  arXiv  Detail & Related papers  (2026-04-19T23:40:26Z)
• Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation [9.540386616651295]
  Chain-of-Thought (CoT) prompting can mitigate hallucinations by encouraging step-by-step reasoning.<n>Our study highlights an overlooked trade-off in the use of reasoning.
  arXiv  Detail & Related papers  (2025-06-20T15:49:37Z)
• HalluLens: LLM Hallucination Benchmark [49.170128733508335]
  Large language models (LLMs) often generate responses that deviate from user input or training data, a phenomenon known as "hallucination"<n>This paper introduces a comprehensive hallucination benchmark, incorporating both new extrinsic and existing intrinsic evaluation tasks.
  arXiv  Detail & Related papers  (2025-04-24T13:40:27Z)
• C-FAITH: A Chinese Fine-Grained Benchmark for Automated Hallucination Evaluation [58.40263551616771]
  We introduce HaluAgent, an agentic framework that automatically constructs fine-grained QA dataset based on some knowledge documents.<n>Our experiments demonstrate that the manually designed rules and prompt optimization can improve the quality of generated data.
  arXiv  Detail & Related papers  (2025-04-14T12:21:55Z)
• Halu-J: Critique-Based Hallucination Judge [22.79828961875684]
  We introduce Halu-J, a critique-based hallucination judge with 7 billion parameters. Halu-J enhances hallucination detection by selecting pertinent evidence and providing detailed critiques. Our experiments indicate that Halu-J outperforms GPT-4o in multiple-evidence hallucination detection.
  arXiv  Detail & Related papers  (2024-07-17T18:21:01Z)
• HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation [19.318217051269382]
  Large Language Models (LLMs) have significantly advanced the field of Natural Language Processing (NLP) HalluDial is the first comprehensive large-scale benchmark for automatic dialogue-level hallucination evaluation. The benchmark includes 4,094 dialogues with a total of 146,856 samples.
  arXiv  Detail & Related papers  (2024-06-11T08:56:18Z)
• Fine-grained Hallucination Detection and Editing for Language Models [109.56911670376932]
  Large language models (LMs) are prone to generate factual errors, which are often called hallucinations. We introduce a comprehensive taxonomy of hallucinations and argue that hallucinations manifest in diverse forms. We propose a novel task of automatic fine-grained hallucination detection and construct a new evaluation benchmark, FavaBench.
  arXiv  Detail & Related papers  (2024-01-12T19:02:48Z)
• Evaluating Hallucinations in Chinese Large Language Models [65.4771562909392]
  We establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. We consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated.
  arXiv  Detail & Related papers  (2023-10-05T07:57:09Z)
• HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models [146.87696738011712]
  Large language models (LLMs) are prone to generate hallucinations, i.e., content that conflicts with the source or cannot be verified by the factual knowledge. To understand what types of content and to which extent LLMs are apt to hallucinate, we introduce the Hallucination Evaluation benchmark for Large Language Models (HaluEval)
  arXiv  Detail & Related papers  (2023-05-19T15:36:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.