CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs

• URL: http://arxiv.org/abs/2505.10496v2
• Date: Fri, 13 Jun 2025 15:39:53 GMT
• Title: CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs
• Abstract summary: CheXGenBench is a rigorous and multifaceted evaluation framework for synthetic chest radiograph generation.<n>It simultaneously assesses fidelity, privacy risks, and clinical utility across state-of-the-art text-to-image generative models.<n>We release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K radiographs generated by the top-performing model.
• Score: 17.95415442789521
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce CheXGenBench, a rigorous and multifaceted evaluation framework for synthetic chest radiograph generation that simultaneously assesses fidelity, privacy risks, and clinical utility across state-of-the-art text-to-image generative models. Despite rapid advancements in generative AI for real-world imagery, medical domain evaluations have been hindered by methodological inconsistencies, outdated architectural comparisons, and disconnected assessment criteria that rarely address the practical clinical value of synthetic samples. CheXGenBench overcomes these limitations through standardised data partitioning and a unified evaluation protocol comprising over 20 quantitative metrics that systematically analyse generation quality, potential privacy vulnerabilities, and downstream clinical applicability across 11 leading text-to-image architectures. Our results reveal critical inefficiencies in the existing evaluation protocols, particularly in assessing generative fidelity, leading to inconsistent and uninformative comparisons. Our framework establishes a standardised benchmark for the medical AI community, enabling objective and reproducible comparisons while facilitating seamless integration of both existing and future generative models. Additionally, we release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K radiographs generated by the top-performing model (Sana 0.6B) in our benchmark to support further research in this critical domain. Through CheXGenBench, we establish a new state-of-the-art and release our framework, models, and SynthCheX-75K dataset at https://raman1121.github.io/CheXGenBench/

Related papers

• Assessment of Conditional Diffusion Model for Synthetic Histopathology Image Generation [13.18104339591425]
  We show that conventional synthetic data evaluation metrics such as Frechet Inception Distance (FID) and Inception Score (IS) may have limitations when applied to histopathology images.<n>To address these limitations, we propose for consideration modified FID and IS approaches utilizing foundation models pretrained on digital pathology datasets.
  arXiv  Detail & Related papers  (2026-08-04T17:51:40Z)
• PathReportEval: A Systematic Benchmark for Pathology Report Generation [5.733136272690028]
  We present a standardized benchmark and evaluation framework for pathology report generation.<n>The framework standardizes preprocessing, feature extraction, training, decoding, and evaluation.<n>A central contribution is the Clinical Report Quality Score (CRQS), a clinically grounded metric for evaluating factual correctness.
  arXiv  Detail & Related papers  (2026-07-20T18:59:47Z)
• BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases [59.19934491064188]
  Artificial intelligence (AI) has achieved remarkable success in medical imaging.<n>These models often perform inconsistently across real-world clinical settings.<n>We develop a benchmark of 85,355 CT scans that systematically evaluates 12 tumor-detection AI models.
  arXiv  Detail & Related papers  (2026-06-23T17:58:59Z)
• Clinically Aware Synthetic Image Generation for Concept Coverage in Chest X-ray Models [3.295369583957252]
  CARS applies targeted perturbations to clinical feature vectors, enabling controlled insertion and deletion of pathological findings.<n>Fine-tuning on CARS-generated images consistently improves precision-recall performance, reduces predictive uncertainty, and improves model calibration.
  arXiv  Detail & Related papers  (2026-03-16T16:48:51Z)
• Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering [94.37535002230504]
  We develop a training-free, inference-time control framework termed Semantically Decoupled Latent Steering.<n>Our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition.<n>We show that our approach significantly reduces the probability of historical hallucinations.
  arXiv  Detail & Related papers  (2026-02-27T04:49:01Z)
• Improving Cardiac Risk Prediction Using Data Generation Techniques [37.94487163156369]
  This work proposes an architecture for the synthesis of realistic clinical records that are coherent with real-world observations.<n>The primary objective is to increase the size and diversity of the available datasets in order to enhance the performance of cardiac risk prediction models.
  arXiv  Detail & Related papers  (2025-12-19T10:17:00Z)
• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• Procrustean Bed for AI-Driven Retrosynthesis: A Unified Framework for Reproducible Evaluation [0.0]
  RetroCast is a unified evaluation suite that standardizes heterogeneous model outputs into a common schema.<n>We evaluate leading search-based and sequence-based algorithms on a new suite of standardized benchmarks.
  arXiv  Detail & Related papers  (2025-12-08T01:26:39Z)
• Benchmarking CXR Foundation Models With Publicly Available MIMIC-CXR and NIH-CXR14 Datasets [0.35441912284181126]
  This work benchmarks two large-scale chest X-ray embedding models (CXR) on public MIMIC-CR and NIH ChestX-ray14 datasets.<n>We extracted embeddings directly from pre-trained encoders, trained lightweight LightGBM classifiers on multiple disease labels, and reported mean AUROC, and F1-score with 95% confidence intervals.
  arXiv  Detail & Related papers  (2025-12-03T12:55:44Z)
• Diffusing the Blind Spot: Uterine MRI Synthesis with Diffusion Models [7.262119921589195]
  We introduce a novel diffusion-based framework for uterine MRI synthesis.<n>Our approach generates anatomically coherent, high fidelity synthetic images that closely mimic real scans.<n>A blinded expert evaluation validates the clinical realism of our synthetic images.
  arXiv  Detail & Related papers  (2025-08-11T12:18:23Z)
• RadFabric: Agentic AI System with Reasoning Capability for Radiology [61.25593938175618]
  RadFabric is a multi agent, multimodal reasoning framework that unifies visual and textual analysis for comprehensive CXR interpretation.<n>System employs specialized CXR agents for pathology detection, an Anatomical Interpretation Agent to map visual findings to precise anatomical structures, and a Reasoning Agent powered by large multimodal reasoning models to synthesize visual, anatomical, and clinical data into transparent and evidence based diagnoses.
  arXiv  Detail & Related papers  (2025-06-17T03:10:33Z)
• Metrics that matter: Evaluating image quality metrics for medical image generation [48.85783422900129]
  This study comprehensively assesses commonly used no-reference image quality metrics using brain MRI data.<n>We evaluate metric sensitivity to a range of challenges, including noise, distribution shifts, and, critically, morphological alterations designed to mimic clinically relevant inaccuracies.
  arXiv  Detail & Related papers  (2025-05-12T01:57:25Z)
• AI-GenBench: A New Ongoing Benchmark for AI-Generated Image Detection [9.540547388707987]
  Ai-GenBench is a novel benchmark designed to address the need for robust detection of AI-generated images in real-world scenarios.<n>By establishing clear evaluation rules and controlled augmentation strategies, Ai-GenBench enables meaningful comparison of detection methods and scalable solutions.
  arXiv  Detail & Related papers  (2025-04-29T15:41:13Z)
• VICCA: Visual Interpretation and Comprehension of Chest X-ray Anomalies in Generated Report Without Human Feedback [1.5839621757142595]
  We propose a novel framework designed to enhance the semantic alignment and localization accuracy of AI-generated medical reports.<n>By comparing features between the original and generated images, we introduce a dual-scoring system.<n>This approach significantly outperforms existing methods, achieving state-of-the-art results in pathology localization and text-to-image alignment.
  arXiv  Detail & Related papers  (2025-01-29T16:02:16Z)
• XGeM: A Multi-Prompt Foundation Model for Multimodal Medical Data Generation [22.908801443059758]
  XGeM is a multimodal generative model designed to support flexible, any-to-any synthesis between medical data modalities.<n>XGeM constructs a shared latent space via contrastive learning and introduces a novel Multi-Prompt Training strategy.<n>We show how XGeM can support key medical data challenges such as anonymization, class imbalance, and data scarcity.
  arXiv  Detail & Related papers  (2025-01-08T16:53:56Z)
• ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation [16.687723916901728]
  We present ReXrank, a leaderboard and challenge for assessing AI-powered radiology report generation. Our framework incorporates ReXGradient, the largest test dataset consisting of 10,000 studies. By providing this standardized evaluation framework, ReXrank enables meaningful comparisons of model performance.
  arXiv  Detail & Related papers  (2024-11-22T18:40:02Z)
• Clinical Evaluation of Medical Image Synthesis: A Case Study in Wireless Capsule Endoscopy [63.39037092484374]
  Synthetic Data Generation based on Artificial Intelligence (AI) can transform the way clinical medicine is delivered.<n>This study focuses on the clinical evaluation of medical SDG, with a proof-of-concept investigation on diagnosing Inflammatory Bowel Disease (IBD) using Wireless Capsule Endoscopy (WCE) images.<n>The results show that TIDE-II generates clinically plausible, very realistic WCE images, of improved quality compared to relevant state-of-the-art generative models.
  arXiv  Detail & Related papers  (2024-10-31T19:48:50Z)
• Towards Predicting Temporal Changes in a Patient's Chest X-ray Images based on Electronic Health Records [9.398163873685798]
  We propose a novel framework, EHRXDiff, which predicts future CXR images by integrating previous CXRs with subsequent medical events.<n>Results show that our framework generates high-quality, realistic future images that effectively capture potential temporal changes.
  arXiv  Detail & Related papers  (2024-09-11T04:49:44Z)
• Revisiting Computer-Aided Tuberculosis Diagnosis [56.80999479735375]
  Tuberculosis (TB) is a major global health threat, causing millions of deaths annually. Computer-aided tuberculosis diagnosis (CTD) using deep learning has shown promise, but progress is hindered by limited training data. We establish a large-scale dataset, namely the Tuberculosis X-ray (TBX11K) dataset, which contains 11,200 chest X-ray (CXR) images with corresponding bounding box annotations for TB areas. This dataset enables the training of sophisticated detectors for high-quality CTD.
  arXiv  Detail & Related papers  (2023-07-06T08:27:48Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.