PathReportEval: A Systematic Benchmark for Pathology Report Generation

• URL: http://arxiv.org/abs/2607.18448v1
• Date: Mon, 20 Jul 2026 18:59:47 GMT
• Title: PathReportEval: A Systematic Benchmark for Pathology Report Generation
• Abstract summary: We present a standardized benchmark and evaluation framework for pathology report generation.<n>The framework standardizes preprocessing, feature extraction, training, decoding, and evaluation.<n>A central contribution is the Clinical Report Quality Score (CRQS), a clinically grounded metric for evaluating factual correctness.
• Score: 5.733136272690028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pathology report generation from whole-slide images (WSIs) is a rapidly growing multimodal learning problem, yet progress is difficult to measure because existing studies use heterogeneous datasets, model settings, visual encoders, and evaluation protocols. Moreover, commonly used natural language generation metrics, including BLEU, ROUGE, and METEOR, primarily reward lexical similarity and often fail to detect clinically consequential errors such as omitted diagnoses, hallucinated findings, or discordant tumor attributes. We present a standardized benchmark and evaluation framework for pathology report generation. The benchmark evaluates four representative methods across three datasets (TCGA, HistAI, and REG 2025) using three pathology foundation encoders (CONCHv1.5, UNI2-h, and H-Optimus-1). Our framework standardizes preprocessing, feature extraction, training, decoding, and evaluation, enabling fair comparison across models while providing a modular platform for integrating new methods, datasets, and encoders. A central contribution is the Clinical Report Quality Score (CRQS), a clinically grounded metric for evaluating factual correctness. CRQS maps reference and generated reports into structured clinical attributes and measures four complementary dimensions: clinical fact coverage, key information recall, hallucination rate, and clinical discordance, producing both an overall score and interpretable sub-scores. Experiments demonstrate that conventional language-generation metrics are weakly aligned with clinical correctness and frequently overestimate report quality. In contrast, CRQS reveals clinically meaningful differences between models and encoders that lexical metrics fail to capture. Together, the benchmark, public plug-and-play framework, and CRQS establish a reproducible foundation for rigorous evaluation of pathology report generation.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Models [46.202482780666635]
  Graph-Hierarchicald Supervised Clinical Alignment reformulates image-report supervision as a hierarchical clinical alignment problem.<n>Our method structures this alignment as a disease-conditioned process, where supervision is decomposed into two levels.<n> Experiments on MIMIC-CXR, IU-Xray, and COV-CTR show that our method consistently improves performance on both conventional and clinical metrics.
  arXiv  Detail & Related papers  (2026-08-25T06:32:26Z)
• Rethinking Clinical Relevance in Chest X-ray Machine Learning: How Evaluation References Define Performance [26.96793950588378]
  Machine learning relies heavily on automated evaluation using reference standards that aim to approximate clinical judgment.<n>We investigate how evaluation-reference choices affect model performance and ranking in both pathology classification and image quality assessment.
  arXiv  Detail & Related papers  (2026-07-28T23:11:04Z)
• SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation [1.5782980044380892]
  Current evaluation protocols for Vision-Language Models in Radiology Report Generation rely on report-level metrics that measure lexical overlap or aggregate clinical correctness.<n>We introduce SHOVIR, a benchmark for evaluating vision shortcut behavior in RRG.
  arXiv  Detail & Related papers  (2026-06-29T12:17:35Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• ReportQA: QA-Based Radiology Report Evaluation [34.29681217890862]
  We propose ReportQA, a clinical-related and flexible radiology report evaluation framework.<n>We first collect datasets covering multiple imaging modalities and anatomical regions.<n>We then construct knowledge trees of clinical entities and attributes with radiologist guidance, and use large language models (LLMs) to extract structured information from raw reports.<n>During evaluation, the report is treated as context, and an LLM acts as a judge model to answer the QA pairs.
  arXiv  Detail & Related papers  (2026-06-13T00:43:03Z)
• CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation [51.11942945171396]
  Conventional evaluation metrics offer only coarse measures of lexical overlap or entity matching.<n>We propose CT-FineBench, a benchmark built from CT-RATE and Merlin to evaluate the fine-grained factual consistency of CT reports.<n>Our benchmark is constructed through a meticulous, Question-Answering (QA) based process.
  arXiv  Detail & Related papers  (2026-04-27T03:32:46Z)
• Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation [10.15221228043609]
  This paper investigates the use of decoding strategies that lead to high aggregate token-overlap scores despite template collapse.<n>We introduce Clinical Association Displacement (CAD), a vocabulary-level framework that quantifies shifts in demographic-based word associations in generated reports.<n>We show that deterministic decoding produces high levels of semantic erasure, while sampling generates diverse outputs but risks introducing new bias.
  arXiv  Detail & Related papers  (2026-03-02T08:59:39Z)
• AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning [73.50200033931148]
  We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists.<n>By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback.<n> Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations.
  arXiv  Detail & Related papers  (2026-01-23T11:59:13Z)
• CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generation [8.08950963137043]
  We present CTest-Metric, a first unified metric assessment framework with three modules determining the clinical feasibility of metrics for CT RRG.<n>The modules test: (i) Writing Style Generalizability (WSG) via LLM-based rephrasing; (ii) Synthetic Error Injection (SEI) at graded severities; and (iii) Metrics-vs-Expert correlation (MvE) using clinician ratings on 175 "disagreement" cases.<n>Eight widely used metrics (BLEU, ROUGE, METEOR, BERTScore-F1, F1-RadGraph, Ra
  arXiv  Detail & Related papers  (2026-01-16T18:09:19Z)
• S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework [39.542375803362965]
  Radiology report generation (RRG) for diagnostic images, such as chest X-rays, plays a pivotal role in both clinical practice and AI.<n>Traditional free-text reports suffer from redundancy and inconsistent language, complicating the extraction of critical clinical details.<n>We present a novel approach to S-RRG that includes dataset construction, model training, and the introduction of a new evaluation framework.
  arXiv  Detail & Related papers  (2025-08-04T05:49:41Z)
• RaTEScore: A Metric for Radiology Report Generation [59.37561810438641]
  This paper introduces a novel, entity-aware metric, as Radiological Report (Text) Evaluation (RaTEScore) RaTEScore emphasizes crucial medical entities such as diagnostic outcomes and anatomical details, and is robust against complex medical synonyms and sensitive to negation expressions. Our evaluations demonstrate that RaTEScore aligns more closely with human preference than existing metrics, validated both on established public benchmarks and our newly proposed RaTE-Eval benchmark.
  arXiv  Detail & Related papers  (2024-06-24T17:49:28Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.