RadOT-Eval: Auditable Structured-Evidence Transport for Radiology Report Evaluation

• URL: http://arxiv.org/abs/2606.08769v1
• Date: Sun, 07 Jun 2026 18:16:56 GMT
• Title: RadOT-Eval: Auditable Structured-Evidence Transport for Radiology Report Evaluation
• Abstract summary: RadOT-Eval is an interpretable structured-evidence optimal transport framework for offline auditing of radiology report generation.<n>It decomposes reference and candidate reports into attribute-structured clinical evidence units.<n>It aligns corresponding evidence using entropy-regularized optimal transport, and uses clinically meaningful side-channel discrepancies in a monotone risk model to predict error burden.
• Score: 10.674070195561761
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic evaluation is critical for high-stakes text generation, where errors often involve omitted findings, hallucinated content, polarity reversals, location changes, uncertainty mismatches, and temporal-comparison errors rather than low surface similarity alone. Radiology report generation provides a challenging test case because generated reports must preserve structured clinical evidence across sources. We present RadOT-Eval, an interpretable structured-evidence optimal transport framework for offline auditing of radiology report generation. RadOT-Eval decomposes reference and candidate reports into attribute-structured clinical evidence units, aligns corresponding evidence using entropy-regularized optimal transport, and uses clinically meaningful side-channel discrepancies in a monotone risk model to predict error burden. All transport, feature, and readout choices are selected using the ReXVal dataset, and the frozen system is evaluated on the independent RadEvalX dataset. RadOT-Eval achieves Spearman correlations of 0.715, 0.548, and 0.399 with total, clinically significant, and clinically insignificant annotated error burden, respectively, yielding higher point estimates than standard evaluation metrics and the open-source large language model (LLM)-based evaluator GREEN-radllama2-7B. In a frozen auxiliary corruption-sensitivity stress test on ReXErr-v1, RadOT-Eval achieves 0.768 AUROC and a 0.990 corrupted-greater-than-clean paired win rate. These results show that structured evidence transport provides an auditable, rank-oriented evaluation tool for high-stakes generated clinical text under ReXVal-only model selection and frozen RadEvalX testing.

Related papers

• RadFusion: Towards Threshold-Controllable Radiology Report Generation [14.362153373103757]
  Existing generation models offer no control over the sensitivity-specificity trade-off of their diagnostic content.<n>We introduce RadFusion, a framework that equips report generation with threshold controllability.
  arXiv  Detail & Related papers  (2026-08-11T05:23:48Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• Detecting Clinical Discrepancies in Health Coaching Agents: A Dual-Stream Memory and Reconciliation Architecture [71.46525715889656]
  General-purpose agent memory systems optimize for coherence by overwriting older facts with the user's latest statement.<n>We introduce a Dual-Stream Memory Architecture that strictly separates the patient narrative from the structured clinical record.<n>We evaluate this architecture on 26 patients across 675 longitudinal wellness coaching sessions.
  arXiv  Detail & Related papers  (2026-04-29T17:59:28Z)
• CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation [51.11942945171396]
  Conventional evaluation metrics offer only coarse measures of lexical overlap or entity matching.<n>We propose CT-FineBench, a benchmark built from CT-RATE and Merlin to evaluate the fine-grained factual consistency of CT reports.<n>Our benchmark is constructed through a meticulous, Question-Answering (QA) based process.
  arXiv  Detail & Related papers  (2026-04-27T03:32:46Z)
• Calibrated Confidence Expression for Radiology Report Generation [33.24673060327421]
  Large Vision-Language Models (LVLMs) in radiology report generation require accurate predictions and clinically interpretable indicators.<n>Current state-of-the-art language models are often overconfident, and research on calibration in multimodal settings such as radiology report generation is limited.<n>We introduce ConRad, a medical reinforcement learning framework for fine-tuning LVLMs to produce verbalized confidence estimates alongside radiology reports.
  arXiv  Detail & Related papers  (2026-03-31T09:37:33Z)
• Ran Score: a LLM-based Evaluation Score for Radiology Report Generation [28.574749773136162]
  We develop a clinician-guided framework combining human expertise and large language models.<n>We show that clinician-guided prompt optimization improves agreement with a radiologist-derived reference standard.
  arXiv  Detail & Related papers  (2026-03-24T08:29:26Z)
• CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation [2.61152955442649]
  CRIMSON is a clinically grounded evaluation framework for chest X-ray report generation.<n>It categorizes errors into a comprehensive taxonomy covering false findings, missing findings, and eight attribute-level errors.<n>CRIMSON is validated through strong alignment with clinically significant error counts annotated by six board-certified radiologists.
  arXiv  Detail & Related papers  (2026-03-06T11:43:42Z)
• Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering [94.37535002230504]
  We develop a training-free, inference-time control framework termed Semantically Decoupled Latent Steering.<n>Our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition.<n>We show that our approach significantly reduces the probability of historical hallucinations.
  arXiv  Detail & Related papers  (2026-02-27T04:49:01Z)
• AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning [73.50200033931148]
  We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists.<n>By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback.<n> Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations.
  arXiv  Detail & Related papers  (2026-01-23T11:59:13Z)
• Automated Structured Radiology Report Generation [11.965406008391371]
  We introduce Structured Radiology Report Generation (SRRG), a new task that reformulates free-text radiology reports into a standardized format.<n>We create a novel dataset by restructuring reports using large language models (LLMs) following strict structured reporting desiderata.<n>We also introduce SRR-BERT, a fine-grained disease classification model trained on 55 labels, enabling more precise and clinically informed evaluation of structured reports.
  arXiv  Detail & Related papers  (2025-05-30T05:23:01Z)
• RaTEScore: A Metric for Radiology Report Generation [59.37561810438641]
  This paper introduces a novel, entity-aware metric, as Radiological Report (Text) Evaluation (RaTEScore) RaTEScore emphasizes crucial medical entities such as diagnostic outcomes and anatomical details, and is robust against complex medical synonyms and sensitive to negation expressions. Our evaluations demonstrate that RaTEScore aligns more closely with human preference than existing metrics, validated both on established public benchmarks and our newly proposed RaTE-Eval benchmark.
  arXiv  Detail & Related papers  (2024-06-24T17:49:28Z)
• Learning to diagnose cirrhosis from radiological and histological labels with joint self and weakly-supervised pretraining strategies [62.840338941861134]
  We propose to leverage transfer learning from large datasets annotated by radiologists, to predict the histological score available on a small annex dataset. We compare different pretraining methods, namely weakly-supervised and self-supervised ones, to improve the prediction of the cirrhosis. This method outperforms the baseline classification of the METAVIR score, reaching an AUC of 0.84 and a balanced accuracy of 0.75.
  arXiv  Detail & Related papers  (2023-02-16T17:06:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.