Privacy-Preserving Chest X-ray Report Generation via Multimodal Federated Learning with ViT and GPT-2

• URL: http://arxiv.org/abs/2505.21715v1
• Date: Tue, 27 May 2025 20:01:12 GMT
• Title: Privacy-Preserving Chest X-ray Report Generation via Multimodal Federated Learning with ViT and GPT-2
• Abstract summary: The study proposes a Multimodal Federated Learning framework for chest X-ray report generation using the IU-Xray dataset.<n>The system utilizes a Vision Transformer (ViT) as the encoder and GPT-2 as the report generator, enabling decentralized training without sharing raw data.<n>The results show that FL can match or surpass centralized models in generating clinically relevant and semantically rich radiology reports.
• Score: 0.1874930567916036
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The automated generation of radiology reports from chest X-ray images holds significant promise in enhancing diagnostic workflows while preserving patient privacy. Traditional centralized approaches often require sensitive data transfer, posing privacy concerns. To address this, the study proposes a Multimodal Federated Learning framework for chest X-ray report generation using the IU-Xray dataset. The system utilizes a Vision Transformer (ViT) as the encoder and GPT-2 as the report generator, enabling decentralized training without sharing raw data. Three Federated Learning (FL) aggregation strategies: FedAvg, Krum Aggregation and a novel Loss-aware Federated Averaging (L-FedAvg) were evaluated. Among these, Krum Aggregation demonstrated superior performance across lexical and semantic evaluation metrics such as ROUGE, BLEU, BERTScore and RaTEScore. The results show that FL can match or surpass centralized models in generating clinically relevant and semantically rich radiology reports. This lightweight and privacy-preserving framework paves the way for collaborative medical AI development without compromising data confidentiality.

Related papers

• HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts [6.917612472314974]
  HarMoE is a dataset-aware mixture-of-experts framework that learns shared cross-dataset medical semantics.<n> Experiments on large-scale chest X-ray benchmarks show that HarMoE consistently improves zero-shot classification transfer.
  arXiv  Detail & Related papers  (2026-08-03T14:00:16Z)
• Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography [75.7789001619107]
  Vision-language pre-training holds great promise for general-purpose medical AI.<n>We propose a tailored framework featuring three key components.<n>A diagnosis-aware prompt strategy employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap.
  arXiv  Detail & Related papers  (2026-06-24T08:24:45Z)
• Grounded Multimodal Retrieval-Augmented Drafting of Radiology Impressions Using Case-Based Similarity Search [0.0]
  We propose a multimodal retrieval-augmented generation (RAG) system for grounded drafting of chest radiograph impressions.<n>The system combines contrastive image-text embeddings, case-based similarity retrieval, and citation-constrained draft generation to ensure factual alignment with historical radiology reports.
  arXiv  Detail & Related papers  (2026-03-18T14:25:50Z)
• Federated Transformer-GNN for Privacy-Preserving Brain Tumor Localization with Modality-Level Explainability [0.2683233968306505]
  We present a federated learning framework for brain tumor localization that enables multi-institutional collaboration without sharing sensitive patient data.<n>Our method extends a hybrid Transformer-Graph Neural Network architecture derived from prior decoder-free supervoxel GNNs.<n>We provide an explainability analysis through Transformer attention mechanisms that reveals which MRI modalities drive the model predictions.
  arXiv  Detail & Related papers  (2026-01-21T14:46:00Z)
• EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation [16.23892817333913]
  EMRRG is a novel X-ray report generation framework that fine-tunes pre-trained Mamba networks.<n>An LLM with a hybrid decoder generates the medical report, enabling end-to-end training and achieving strong results on benchmark datasets.
  arXiv  Detail & Related papers  (2025-10-19T09:54:36Z)
• Adapting HFMCA to Graph Data: Self-Supervised Learning for Generalizable fMRI Representations [57.054499278843856]
  Functional magnetic resonance imaging (fMRI) analysis faces significant challenges due to limited dataset sizes and domain variability between studies.<n>Traditional self-supervised learning methods inspired by computer vision often rely on positive and negative sample pairs.<n>We propose adapting a recently developed Hierarchical Functional Maximal Correlation Algorithm (HFMCA) to graph-structured fMRI data.
  arXiv  Detail & Related papers  (2025-10-05T12:35:01Z)
• Exploring the Capabilities of LLM Encoders for Image-Text Retrieval in Chest X-rays [8.019362739504087]
  Vision-language pretraining has advanced image-text alignment, yet progress in radiology remains constrained by the heterogeneity of clinical reports.<n>We ask whether large language model (LLM) encoders can provide robust clinical representations that transfer across diverse styles.<n>We introduce LLM2VEC4CXR, a domain-adapted encoder for chest X-ray reports, and LLM2CLIP4CXR, a dual-tower framework that couples this encoder with a vision backbone.
  arXiv  Detail & Related papers  (2025-09-17T09:44:59Z)
• PriorRG: Prior-Guided Contrastive Pre-training and Coarse-to-Fine Decoding for Chest X-ray Report Generation [12.860257420677122]
  PriorRG is a chest X-ray report generation framework that emulates real-world clinical via a two-stage training pipeline.<n>In Stage 1, we introduce a prior-guided contrast pre-training scheme that leverages clinical context guidetemporal feature extraction.<n>In Stage 2, we integrate a prior-aware coarsetemporal decoding for generation that enhances prior knowledge with the vision encoders hidden states.
  arXiv  Detail & Related papers  (2025-08-07T13:02:20Z)
• Decentralized LoRA Augmented Transformer with Context-aware Multi-scale Feature Learning for Secured Eye Diagnosis [2.1358421658740214]
  This paper proposes a novel Data efficient Image Transformer (DeiT) based framework that integrates context aware multiscale patch embedding, Low-Rank Adaptation (LoRA), knowledge distillation, and federated learning to address these challenges in a unified manner.<n>The proposed model effectively captures both local and global retinal features by leveraging multi scale patch representations with local and global attention mechanisms.
  arXiv  Detail & Related papers  (2025-05-11T13:51:56Z)
• Reason Like a Radiologist: Chain-of-Thought and Reinforcement Learning for Verifiable Report Generation [13.580272788409092]
  BoxMed-RL is a groundbreaking unified training framework for generating spatially verifiable and explainable radiology reports.<n>Built on a large vision-language model, BoxMed-RL revolutionizes report generation through two integrated phases.<n>BoxMed-RL achieves an average 7% improvement in both METEOR and ROUGE-L metrics compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-25T16:05:06Z)
• The Devil is in the Prompts: De-Identification Traces Enhance Memorization Risks in Synthetic Chest X-Ray Generation [0.0]
  Generative models, particularly text-to-image (T2I) diffusion models, play a crucial role in medical image analysis.<n>These models are prone to training data memorization, posing significant risks to patient privacy.<n>This study presents the first systematic attempt to identify prompts that contribute the most to training data memorization.
  arXiv  Detail & Related papers  (2025-02-11T12:36:00Z)
• Federated Learning for Coronary Artery Plaque Detection in Atherosclerosis Using IVUS Imaging: A Multi-Hospital Collaboration [8.358846277772779]
  Traditional interpretation of Intravascular Ultrasound (IVUS) images during Percutaneous Coronary Intervention ( PCI) is time-intensive and inconsistent.<n>A parallel 2D U-Net model with a multi-stage segmentation architecture has been developed to enable secure data analysis across institutions.<n>A Dice Similarity Coefficient (DSC) of 0.706, the model effectively identifies plaques and detects circular boundaries in real-time.
  arXiv  Detail & Related papers  (2024-12-19T13:06:28Z)
• FeDETR: a Federated Approach for Stenosis Detection in Coronary Angiography [0.3823356975862007]
  coronary stenosis is an underlying factor in heart failure. Current practice for grading coronary lesions suffers from several drawbacks. Deep learning methods have emerged to assist cardiologists in automating the estimation of FFR/iFR values.
  arXiv  Detail & Related papers  (2024-09-21T23:52:05Z)
• Summarizing Radiology Reports Findings into Impressions [1.8964110318127383]
  We present a model with state-of-art radiology report summarization performance. We also provide an analysis of the model limitations and radiology knowledge gain. Our best performing model was a fine-tuned BERT-to-BERT encoder-decoder with 58.75/100 ROUGE-L F1.
  arXiv  Detail & Related papers  (2024-05-10T20:29:25Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• Improving Multiple Sclerosis Lesion Segmentation Across Clinical Sites: A Federated Learning Approach with Noise-Resilient Training [75.40980802817349]
  Deep learning models have shown promise for automatically segmenting MS lesions, but the scarcity of accurately annotated data hinders progress in this area. We introduce a Decoupled Hard Label Correction (DHLC) strategy that considers the imbalanced distribution and fuzzy boundaries of MS lesions. We also introduce a Centrally Enhanced Label Correction (CELC) strategy, which leverages the aggregated central model as a correction teacher for all sites.
  arXiv  Detail & Related papers  (2023-08-31T00:36:10Z)
• Source-Free Collaborative Domain Adaptation via Multi-Perspective Feature Enrichment for Functional MRI Analysis [55.03872260158717]
  Resting-state MRI functional (rs-fMRI) is increasingly employed in multi-site research to aid neurological disorder analysis. Many methods have been proposed to reduce fMRI heterogeneity between source and target domains. But acquiring source data is challenging due to concerns and/or data storage burdens in multi-site studies. We design a source-free collaborative domain adaptation framework for fMRI analysis, where only a pretrained source model and unlabeled target data are accessible.
  arXiv  Detail & Related papers  (2023-08-24T01:30:18Z)
• Medical Image Captioning via Generative Pretrained Transformers [57.308920993032274]
  We combine two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The proposed model is tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO.
  arXiv  Detail & Related papers  (2022-09-28T10:27:10Z)
• Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation [116.87918100031153]
  We propose a Cross-modal clinical Graph Transformer (CGT) for ophthalmic report generation (ORG) CGT injects clinical relation triples into the visual features as prior knowledge to drive the decoding procedure. Experiments on the large-scale FFA-IR benchmark demonstrate that the proposed CGT is able to outperform previous benchmark methods.
  arXiv  Detail & Related papers  (2022-06-04T13:16:30Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.