RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

• URL: http://arxiv.org/abs/2609.01470v1
• Date: Tue, 01 Sep 2026 16:05:17 GMT
• Title: RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching
• Abstract summary: RadMatch is a metric that decomposes report comparison into a structured finding-level matching with significance-aware scoring and error characterization.<n>It is the most clinically aligned metric, matching inter-radiologist agreement on ReXVal and more than doubling the best prior metric on the harder RadEvalExpert.<n>We will release RadMatch as open-source code with an interactive dashboard for inspecting results.
• Score: 5.0049748707447
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language model (LLM)-based metrics are now the best-correlated with radiologist judgment, yet they output a single opaque score that neither a clinician nor a model builder can easily interpret or audit. We introduce RadMatch, a multi-stage, LLM-based metric that decomposes report comparison into a structured finding-level matching with significance-aware scoring and error characterization across seven clinical attribute dimensions (status, location, severity, morphology, certainty, longitudinal comparison, and measurement). The main score is the actionable-error count, both interpretable and auditable. Candidate findings are graded correct, partial, or incorrect, and unmatched findings are counted as missed or hallucinated. Triage and actionable safety recall/precision and per-subset views add complementary, deployment-oriented lenses. Across two expert benchmarks, RadMatch is the most clinically aligned metric, matching inter-radiologist agreement on ReXVal and more than doubling the best prior metric on the harder RadEvalExpert. Relying only on few-shot prompting, it is designed to extend to other modalities and anatomies. We will release RadMatch as open-source code with an interactive dashboard for inspecting results.

Related papers

• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation [51.11942945171396]
  Conventional evaluation metrics offer only coarse measures of lexical overlap or entity matching.<n>We propose CT-FineBench, a benchmark built from CT-RATE and Merlin to evaluate the fine-grained factual consistency of CT reports.<n>Our benchmark is constructed through a meticulous, Question-Answering (QA) based process.
  arXiv  Detail & Related papers  (2026-04-27T03:32:46Z)
• Calibrated Confidence Expression for Radiology Report Generation [33.24673060327421]
  Large Vision-Language Models (LVLMs) in radiology report generation require accurate predictions and clinically interpretable indicators.<n>Current state-of-the-art language models are often overconfident, and research on calibration in multimodal settings such as radiology report generation is limited.<n>We introduce ConRad, a medical reinforcement learning framework for fine-tuning LVLMs to produce verbalized confidence estimates alongside radiology reports.
  arXiv  Detail & Related papers  (2026-03-31T09:37:33Z)
• CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation [2.61152955442649]
  CRIMSON is a clinically grounded evaluation framework for chest X-ray report generation.<n>It categorizes errors into a comprehensive taxonomy covering false findings, missing findings, and eight attribute-level errors.<n>CRIMSON is validated through strong alignment with clinically significant error counts annotated by six board-certified radiologists.
  arXiv  Detail & Related papers  (2026-03-06T11:43:42Z)
• Semantic Similarity in Radiology Reports via LLMs and NER [1.2489632787815885]
  Radiology report evaluation is a crucial part of radiologists' training and plays a key role in ensuring diagnostic accuracy.<n>Identifying semantic differences between preliminary and final reports is essential for junior doctors, both as a training tool and to help uncover gaps in clinical knowledge.<n>While AI in radiology is a rapidly growing field, the application of large language models (LLMs) remains challenging due to the need for specialised domain knowledge.<n>We propose Llama-EntScore, a semantic similarity scoring method using a combination of Llama 3.1 and NER with tunable weights to emphasise or de-emphasise specific types of differences.
  arXiv  Detail & Related papers  (2025-10-03T15:31:11Z)
• RadEval: A framework for radiology text evaluation [18.848190941379222]
  RadEval is a unified, open-source framework for evaluating radiology texts.<n>It consolidates a diverse range of metrics, from classic n-gram overlap to clinical concept-based scores.<n>We release a richly annotated expert dataset with over 450 clinically significant error labels.
  arXiv  Detail & Related papers  (2025-09-22T17:03:48Z)
• Ontology-Based Concept Distillation for Radiology Report Retrieval and Labeling [10.504309161945065]
  Most existing methods rely on comparing high-dimensional text embeddings from models like CLIP or CXR-BERT.<n>We propose a novel, ontology-driven alternative for comparing radiology report texts based on clinically grounded concepts from the Unified Medical Language System.<n>Our method extracts standardised medical entities from free-text reports using an enhanced pipeline built on RadGraph-XL and SapBERT.
  arXiv  Detail & Related papers  (2025-08-27T14:20:50Z)
• Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation [32.410641778559544]
  ICARE (Interpretable and Clinically-grounded Agent-based Report Evaluation) is an interpretable evaluation framework.<n>Two agents, each with either the ground-truth or generated report, generate clinically meaningful questions and quiz each other.<n>By linking scores to question-answer pairs, ICARE enables transparent, and interpretable assessment.
  arXiv  Detail & Related papers  (2025-08-04T18:28:03Z)
• RaTEScore: A Metric for Radiology Report Generation [59.37561810438641]
  This paper introduces a novel, entity-aware metric, as Radiological Report (Text) Evaluation (RaTEScore) RaTEScore emphasizes crucial medical entities such as diagnostic outcomes and anatomical details, and is robust against complex medical synonyms and sensitive to negation expressions. Our evaluations demonstrate that RaTEScore aligns more closely with human preference than existing metrics, validated both on established public benchmarks and our newly proposed RaTE-Eval benchmark.
  arXiv  Detail & Related papers  (2024-06-24T17:49:28Z)
• End-to-End Page-Level Assessment of Handwritten Text Recognition [69.55992406968495]
  HTR systems increasingly face the end-to-end page-level transcription of a document. Standard metrics do not take into account the inconsistencies that might appear. We propose a two-fold evaluation, where the transcription accuracy and the RO goodness are considered separately.
  arXiv  Detail & Related papers  (2023-01-14T15:43:07Z)
• COMPOSE: Cross-Modal Pseudo-Siamese Network for Patient Trial Matching [70.08786840301435]
  We propose CrOss-Modal PseudO-SiamEse network (COMPOSE) to address these challenges for patient-trial matching. Experiment results show COMPOSE can reach 98.0% AUC on patient-criteria matching and 83.7% accuracy on patient-trial matching.
  arXiv  Detail & Related papers  (2020-06-15T21:01:33Z)
• DeepEnroll: Patient-Trial Matching with Deep Embedding and Entailment Prediction [67.91606509226132]
  Clinical trials are essential for drug development but often suffer from expensive, inaccurate and insufficient patient recruitment. DeepEnroll is a cross-modal inference learning model to jointly encode enrollment criteria (tabular data) into a shared latent space for matching inference.
  arXiv  Detail & Related papers  (2020-01-22T17:51:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.