Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging

• URL: http://arxiv.org/abs/2010.03060v5
• Date: Thu, 9 Sep 2021 00:25:53 GMT
• Title: Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging
• Abstract summary: We propose a form of weak supervision to improve the image interpretation performance of a neural network. We use an image-text matching task to train a feature extractor and then fine-tune it in a transfer learning setting for a supervised task. The end result is a neural network that automatically interprets imagery without requiring textual reports during inference.
• Score: 17.530514174860826
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A key challenge in training neural networks for a given medical imaging task is often the difficulty of obtaining a sufficient number of manually labeled examples. In contrast, textual imaging reports, which are often readily available in medical records, contain rich but unstructured interpretations written by experts as part of standard clinical practice. We propose using these textual reports as a form of weak supervision to improve the image interpretation performance of a neural network without requiring additional manually labeled examples. We use an image-text matching task to train a feature extractor and then fine-tune it in a transfer learning setting for a supervised task using a small labeled dataset. The end result is a neural network that automatically interprets imagery without requiring textual reports during inference. This approach can be applied to any task for which text-image pairs are readily available. We evaluate our method on three classification tasks and find consistent performance improvements, reducing the need for labeled data by 67%-98%.

Related papers

• A Human-in-the-Loop Framework for Efficient Prompt Selection in Microscopy Vision-Language Models [1.5824473848781937]
  Deep-learning pipelines for microscopy image classification often require expensive, labor- and time-intensive expert annotation to produce high-quality ground truth for training.<n>Recent work has shown that prompt tuning of vision-language models can reduce manual annotation by constructing a small prompt set of expert-verified image-caption exemplars.
  arXiv  Detail & Related papers  (2026-05-19T21:02:43Z)
• When are radiology reports useful for training medical image classifiers? [17.157946138233758]
  We conduct a systematic study of how radiology reports can be used during both pre-training and fine-tuning.<n>Our findings reveal that pre-training through explicit image-text alignment can be detrimental in settings where it's not.<n>These results provide actionable insights into when and how to leverage privileged text data to train medical image classifiers.
  arXiv  Detail & Related papers  (2025-10-28T13:01:42Z)
• Learning Generalized Medical Image Representations through Image-Graph Contrastive Pretraining [11.520404630575749]
  We develop an Image-Graph Contrastive Learning framework that pairs chest X-rays with structured report knowledge graphs automatically extracted from radiology notes. Our approach uniquely encodes the disconnected graph components via a relational graph convolution network and transformer attention.
  arXiv  Detail & Related papers  (2024-05-15T12:27:38Z)
• MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning [20.33625985769796]
  Existing contrastive language-image pre-training aims to learn a joint representation by matching abundant image-text pairs. We propose a Medical Language-Image Pre-training framework, which exploits the limited image-text medical data more efficiently. Our evaluation results show that MLIP outperforms previous work in zero/few-shot classification and few-shot segmentation tasks by a large margin.
  arXiv  Detail & Related papers  (2024-01-03T07:54:13Z)
• Unified Medical Image-Text-Label Contrastive Learning With Continuous Prompt [3.218449686637963]
  We propose a unified Image-Text-Label contrastive learning framework based on continuous prompts. We demonstrate through sufficient experiments that the Unified Medical Contrastive Learning framework exhibits excellent performance on several downstream tasks.
  arXiv  Detail & Related papers  (2023-07-12T05:19:10Z)
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing [53.89917396428747]
  Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. We explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model.
  arXiv  Detail & Related papers  (2023-01-11T16:35:33Z)
• Universal Multimodal Representation for Language Understanding [110.98786673598015]
  This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively.
  arXiv  Detail & Related papers  (2023-01-09T13:54:11Z)
• Self-Supervised Learning as a Means To Reduce the Need for Labeled Data in Medical Image Analysis [64.4093648042484]
  We use a dataset of chest X-ray images with bounding box labels for 13 different classes of anomalies. We show that it is possible to achieve similar performance to a fully supervised model in terms of mean average precision and accuracy with only 60% of the labeled data.
  arXiv  Detail & Related papers  (2022-06-01T09:20:30Z)
• Robust Medical Image Classification from Noisy Labeled Data with Global and Local Representation Guided Co-training [73.60883490436956]
  We propose a novel collaborative training paradigm with global and local representation learning for robust medical image classification. We employ the self-ensemble model with a noisy label filter to efficiently select the clean and noisy samples. We also design a novel global and local representation learning scheme to implicitly regularize the networks to utilize noisy samples.
  arXiv  Detail & Related papers  (2022-05-10T07:50:08Z)
• Cascaded Robust Learning at Imperfect Labels for Chest X-ray Segmentation [61.09321488002978]
  We present a novel cascaded robust learning framework for chest X-ray segmentation with imperfect annotation. Our model consists of three independent network, which can effectively learn useful information from the peer networks. Our methods could achieve a significant improvement on the accuracy in segmentation tasks compared to the previous methods.
  arXiv  Detail & Related papers  (2021-04-05T15:50:16Z)
• Contrastive Learning of Medical Visual Representations from Paired Images and Text [38.91117443316013]
  We propose ConVIRT, an unsupervised strategy to learn medical visual representations by exploiting naturally occurring descriptive paired text. Our new method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input.
  arXiv  Detail & Related papers  (2020-10-02T02:10:18Z)
• Multi-label Thoracic Disease Image Classification with Cross-Attention Networks [65.37531731899837]
  We propose a novel scheme of Cross-Attention Networks (CAN) for automated thoracic disease classification from chest x-ray images. We also design a new loss function that beyond cross-entropy loss to help cross-attention process and is able to overcome the imbalance between classes and easy-dominated samples within each class.
  arXiv  Detail & Related papers  (2020-07-21T14:37:00Z)
• Train, Learn, Expand, Repeat [0.15833270109954134]
  High-quality labeled data is essential to successfully train supervised machine learning models. Medical professionals who can expertly label the data are a scarce and expensive resource. We apply this technique on the segmentation of intracranial hemorrhage (ICH) in CT scans of the brain.
  arXiv  Detail & Related papers  (2020-03-18T20:55:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.