Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders

• URL: http://arxiv.org/abs/2212.07050v1
• Date: Wed, 14 Dec 2022 06:04:18 GMT
• Title: Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders
• Authors: Jongseong Jang, Daeun Kyung, Seung Hwan Kim, Honglak Lee, Kyunghoon Bae, Edward Choi
• Abstract summary: We propose a new fine-tuning strategy based on sentence sampling and positive-pair loss relaxation for improving the downstream zero-shot pathology classification performance. Our method consistently showed dramatically improved zero-shot pathology classification performance on four different chest X-ray datasets.
• Score: 51.14431540035141
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deep neural networks have been successfully adopted to diverse domains including pathology classification based on medical images. However, large-scale and high-quality data to train powerful neural networks are rare in the medical domain as the labeling must be done by qualified experts. Researchers recently tackled this problem with some success by taking advantage of models pre-trained on large-scale general domain data. Specifically, researchers took contrastive image-text encoders (e.g., CLIP) and fine-tuned it with chest X-ray images and paired reports to perform zero-shot pathology classification, thus completely removing the need for pathology-annotated images to train a classification model. Existing studies, however, fine-tuned the pre-trained model with the same contrastive learning objective, and failed to exploit the multi-labeled nature of medical image-report pairs. In this paper, we propose a new fine-tuning strategy based on sentence sampling and positive-pair loss relaxation for improving the downstream zero-shot pathology classification performance, which can be applied to any pre-trained contrastive image-text encoders. Our method consistently showed dramatically improved zero-shot pathology classification performance on four different chest X-ray datasets and 3 different pre-trained models (5.77% average AUROC increase). In particular, fine-tuning CLIP with our method showed much comparable or marginally outperformed to board-certified radiologists (0.619 vs 0.625 in F1 score and 0.530 vs 0.544 in MCC) in zero-shot classification of five prominent diseases from the CheXpert dataset.

Related papers

• Chest Disease Detection In X-Ray Images Using Deep Learning Classification Method [0.0]
  We investigate the performance across multiple classification models to classify chest X-ray images.<n>We fine-tuned these pre-trained architectures on a labeled medical x-ray images.<n>The initial results are promising with high accuracy and strong performance in key classification metrics.
  arXiv  Detail & Related papers  (2025-05-28T17:24:33Z)
• Chest X-ray Classification using Deep Convolution Models on Low-resolution images with Uncertain Labels [3.038642416291856]
  We report classification results by experimenting on different input image sizes of Chest X-rays to deep CNN models. We use an ensemble of multi-label classification models on frontal and lateral studies. For pathologies Cardiomegaly, Consolidation and Edema, we obtain 3% higher accuracy with our model architecture.
  arXiv  Detail & Related papers  (2025-04-12T01:13:00Z)
• Trustworthy image-to-image translation: evaluating uncertainty calibration in unpaired training scenarios [0.0]
  Mammographic screening is an effective method for detecting breast cancer, facilitating early diagnosis. Deep neural networks have been shown effective in some studies, but their tendency to overfit leaves considerable risk for poor generalisation and misdiagnosis. Data augmentation schemes based on unpaired neural style transfer models have been proposed that improve generalisability. We evaluate their performance when trained on image patches parsed from three open access mammography datasets and one non-medical image dataset.
  arXiv  Detail & Related papers  (2025-01-29T11:09:50Z)
• Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime [70.04389979779195]
  This paper explores training medical vision-language models (VLMs) where the visual and language inputs are embedded into a common space. We explore several candidate methods to improve low-data performance, including adapting generic pre-trained models to novel image and text domains. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports.
  arXiv  Detail & Related papers  (2023-03-30T18:20:00Z)
• RadTex: Learning Efficient Radiograph Representations from Text Reports [7.090896766922791]
  We build a data-efficient learning framework that utilizes radiology reports to improve medical image classification performance with limited labeled data. Our model achieves higher classification performance than ImageNet-supervised pretraining when labeled training data is limited.
  arXiv  Detail & Related papers  (2022-08-05T15:06:26Z)
• Application of Transfer Learning and Ensemble Learning in Image-level Classification for Breast Histopathology [9.037868656840736]
  In Computer-Aided Diagnosis (CAD), traditional classification models mostly use a single network to extract features. This paper proposes a deep ensemble model based on image-level labels for the binary classification of benign and malignant lesions. Result: In the ensemble network model with accuracy as the weight, the image-level binary classification achieves an accuracy of $98.90%$.
  arXiv  Detail & Related papers  (2022-04-18T13:31:53Z)
• A Deep Learning Technique using a Sequence of Follow Up X-Rays for Disease classification [3.3345134768053635]
  The ability to predict lung and heart based diseases using deep learning techniques is central to many researchers. We present a hypothesis that X-rays of patients included with the follow up history of their most recent three chest X-ray images would perform better in disease classification.
  arXiv  Detail & Related papers  (2022-03-28T19:58:47Z)
• Incremental Cross-Domain Adaptation for Robust Retinopathy Screening via Bayesian Deep Learning [7.535751594024775]
  Retinopathy represents a group of retinal diseases that, if not treated timely, can cause severe visual impairments or even blindness. This paper presents a novel incremental cross-domain adaptation instrument that allows any deep classification model to progressively learn abnormal retinal pathologies. The proposed framework, evaluated on six public datasets, outperforms the state-of-the-art competitors by achieving an overall accuracy and F1 score of 0.9826 and 0.9846, respectively.
  arXiv  Detail & Related papers  (2021-10-18T13:45:21Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• Multi-label Thoracic Disease Image Classification with Cross-Attention Networks [65.37531731899837]
  We propose a novel scheme of Cross-Attention Networks (CAN) for automated thoracic disease classification from chest x-ray images. We also design a new loss function that beyond cross-entropy loss to help cross-attention process and is able to overcome the imbalance between classes and easy-dominated samples within each class.
  arXiv  Detail & Related papers  (2020-07-21T14:37:00Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.