Benchmarking Robustness of Contrastive Learning Models for Medical Image-Report Retrieval

• URL: http://arxiv.org/abs/2501.09134v1
• Date: Wed, 15 Jan 2025 20:37:04 GMT
• Title: Benchmarking Robustness of Contrastive Learning Models for Medical Image-Report Retrieval
• Authors: Demetrio Deanda, Yuktha Priya Masupalli, Jeong Yang, Young Lee, Zechun Cao, Gongbo Liang,
• Abstract summary: This study benchmarks the robustness of four state-of-the-art contrastive learning models: CLIP, CXR-RePaiR, MedCLIP, and CXR-CLIP. Our findings reveal that all evaluated models are highly sensitive to out-of-distribution data. By addressing these limitations, we can develop more reliable cross-domain retrieval models for medical applications.
• Score: 2.9801426627439453
• License:
• Abstract: Medical images and reports offer invaluable insights into patient health. The heterogeneity and complexity of these data hinder effective analysis. To bridge this gap, we investigate contrastive learning models for cross-domain retrieval, which associates medical images with their corresponding clinical reports. This study benchmarks the robustness of four state-of-the-art contrastive learning models: CLIP, CXR-RePaiR, MedCLIP, and CXR-CLIP. We introduce an occlusion retrieval task to evaluate model performance under varying levels of image corruption. Our findings reveal that all evaluated models are highly sensitive to out-of-distribution data, as evidenced by the proportional decrease in performance with increasing occlusion levels. While MedCLIP exhibits slightly more robustness, its overall performance remains significantly behind CXR-CLIP and CXR-RePaiR. CLIP, trained on a general-purpose dataset, struggles with medical image-report retrieval, highlighting the importance of domain-specific training data. The evaluation of this work suggests that more effort needs to be spent on improving the robustness of these models. By addressing these limitations, we can develop more reliable cross-domain retrieval models for medical applications.

Related papers

• Fairness Analysis of CLIP-Based Foundation Models for X-Ray Image Classification [15.98427699337596]
  We perform a comprehensive fairness analysis of CLIP-like models applied to X-ray image classification. We assess their performance and fairness across diverse patient demographics and disease categories using zero-shot inference and various fine-tuning techniques.
  arXiv  Detail & Related papers  (2025-01-31T12:23:50Z)
• MAPUNetR: A Hybrid Vision Transformer and U-Net Architecture for Efficient and Interpretable Medical Image Segmentation [0.0]
  We introduce MAPUNetR, a novel architecture that synergizes the strengths of transformer models with the proven U-Net framework for medical image segmentation. Our model addresses the resolution preservation challenge and incorporates attention maps highlighting segmented regions, increasing accuracy and interpretability. Our experiments show that the model maintains stable performance and potential as a powerful tool for medical image segmentation in clinical practice.
  arXiv  Detail & Related papers  (2024-10-29T16:52:57Z)
• Cross-model Mutual Learning for Exemplar-based Medical Image Segmentation [25.874281336821685]
  Cross-model Mutual learning framework for Exemplar-based Medical image (CMEMS) We introduce a novel Cross-model Mutual learning framework for Exemplar-based Medical image (CMEMS)
  arXiv  Detail & Related papers  (2024-04-18T00:18:07Z)
• Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime [70.04389979779195]
  This paper explores training medical vision-language models (VLMs) where the visual and language inputs are embedded into a common space. We explore several candidate methods to improve low-data performance, including adapting generic pre-trained models to novel image and text domains. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports.
  arXiv  Detail & Related papers  (2023-03-30T18:20:00Z)
• Does image resolution impact chest X-ray based fine-grained Tuberculosis-consistent lesion segmentation? [3.3086274755158325]
  Deep learning models are reportedly trained on reduced image resolutions citing reasons for the lack of computational resources. This study investigated performance gains achieved through training an Inception-V3-based UNet model using various image/mask resolutions.
  arXiv  Detail & Related papers  (2023-01-10T15:34:39Z)
• Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders [50.689585476660554]
  We propose a new fine-tuning strategy that includes positive-pair loss relaxation and random sentence sampling. Our approach consistently improves overall zero-shot pathology classification across four chest X-ray datasets and three pre-trained models.
  arXiv  Detail & Related papers  (2022-12-14T06:04:18Z)
• Competence-based Multimodal Curriculum Learning for Medical Report Generation [98.10763792453925]
  We propose a Competence-based Multimodal Curriculum Learning framework ( CMCL) to alleviate the data bias and make best use of available data. Specifically, CMCL simulates the learning process of radiologists and optimize the model in a step by step manner. Experiments on the public IU-Xray and MIMIC-CXR datasets show that CMCL can be incorporated into existing models to improve their performance.
  arXiv  Detail & Related papers  (2022-06-24T08:16:01Z)
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy [70.71457102672545]
  We compare the impact of different training procedures for diabetic retinopathy grading. We investigate different aspects such as quantitative performance, statistics of the learned feature representations, interpretability and robustness to image distortions. Our results indicate that models from ImageNet pretraining report a significant increase in performance, generalization and robustness to image distortions.
  arXiv  Detail & Related papers  (2021-06-25T08:32:45Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• Deep Metric Learning-based Image Retrieval System for Chest Radiograph and its Clinical Applications in COVID-19 [12.584626589565522]
  Chest radiograph (CXR) has been playing a crucial role in COVID-19 patient triaging, diagnosing and monitoring. Considering the mixed and unspecific signals in CXR, an image retrieval model of CXR that provides both similar images and associated clinical information can be more clinically meaningful. In this work we develop a novel CXR image retrieval model based on deep metric learning.
  arXiv  Detail & Related papers  (2020-11-26T03:16:48Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.