Learning Generalized Medical Image Representations through Image-Graph Contrastive Pretraining

• URL: http://arxiv.org/abs/2405.09594v1
• Date: Wed, 15 May 2024 12:27:38 GMT
• Title: Learning Generalized Medical Image Representations through Image-Graph Contrastive Pretraining
• Authors: Sameer Khanna, Daniel Michael, Marinka Zitnik, Pranav Rajpurkar,
• Abstract summary: We develop an Image-Graph Contrastive Learning framework that pairs chest X-rays with structured report knowledge graphs automatically extracted from radiology notes. Our approach uniquely encodes the disconnected graph components via a relational graph convolution network and transformer attention.
• Score: 11.520404630575749
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Medical image interpretation using deep learning has shown promise but often requires extensive expert-annotated datasets. To reduce this annotation burden, we develop an Image-Graph Contrastive Learning framework that pairs chest X-rays with structured report knowledge graphs automatically extracted from radiology notes. Our approach uniquely encodes the disconnected graph components via a relational graph convolution network and transformer attention. In experiments on the CheXpert dataset, this novel graph encoding strategy enabled the framework to outperform existing methods that use image-text contrastive learning in 1% linear evaluation and few-shot settings, while achieving comparable performance to radiologists. By exploiting unlabeled paired images and text, our framework demonstrates the potential of structured clinical insights to enhance contrastive learning for medical images. This work points toward reducing demands on medical experts for annotations, improving diagnostic precision, and advancing patient care through robust medical image understanding.

Related papers

• MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning [48.97640824497327]
  We propose a novel framework leveraging domain-specific medical knowledge as guiding signals to integrate language information into the visual domain through image-text contrastive learning. Our model includes global contrastive learning with our designed divergence encoder, local token-knowledge-patch alignment contrastive learning, and knowledge-guided category-level contrastive learning with expert knowledge. Notably, MLIP surpasses state-of-the-art methods even with limited annotated data, highlighting the potential of multimodal pre-training in advancing medical representation learning.
  arXiv  Detail & Related papers  (2024-02-03T05:48:50Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation [92.73584302508907]
  We propose a knowledge graph with Dynamic structure and nodes to facilitate medical report generation with Contrastive Learning. In detail, the fundamental structure of our graph is pre-constructed from general knowledge. Each image feature is integrated with its very own updated graph before being fed into the decoder module for report generation.
  arXiv  Detail & Related papers  (2023-03-18T03:53:43Z)
• Representative Image Feature Extraction via Contrastive Learning Pretraining for Chest X-ray Report Generation [19.69560434388278]
  The goal of medical report generation is to accurately capture and describe the image findings. Previous works pretrain their visual encoding neural networks with large datasets in different domains. We propose a framework that uses a contrastive learning approach to pretrain the visual encoder and requires no additional meta information.
  arXiv  Detail & Related papers  (2022-09-04T12:07:19Z)
• RadTex: Learning Efficient Radiograph Representations from Text Reports [7.090896766922791]
  We build a data-efficient learning framework that utilizes radiology reports to improve medical image classification performance with limited labeled data. Our model achieves higher classification performance than ImageNet-supervised pretraining when labeled training data is limited.
  arXiv  Detail & Related papers  (2022-08-05T15:06:26Z)
• Self-Ensembling Contrastive Learning for Semi-Supervised Medical Image Segmentation [6.889911520730388]
  We aim to boost the performance of semi-supervised learning for medical image segmentation with limited labels. We learn latent representations directly at feature-level by imposing contrastive loss on unlabeled images. We conduct experiments on an MRI and a CT segmentation dataset and demonstrate that the proposed method achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-05-27T03:27:58Z)
• Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
  We propose a deep co-attention network for multi-view subspace learning. It aims to extract both the common information and the complementary information in an adversarial setting. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
  arXiv  Detail & Related papers  (2021-02-15T18:46:44Z)
• Contrastive Learning of Medical Visual Representations from Paired Images and Text [38.91117443316013]
  We propose ConVIRT, an unsupervised strategy to learn medical visual representations by exploiting naturally occurring descriptive paired text. Our new method of pretraining medical image encoders with the paired text data via a bidirectional contrastive objective between the two modalities is domain-agnostic, and requires no additional expert input.
  arXiv  Detail & Related papers  (2020-10-02T02:10:18Z)
• Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation [107.3538598876467]
  We propose an Auxiliary Signal-Guided Knowledge-Decoder (ASGK) to mimic radiologists' working patterns. ASGK integrates internal visual feature fusion and external medical linguistic information to guide medical knowledge transfer and learning.
  arXiv  Detail & Related papers  (2020-06-06T01:00:15Z)
• Dynamic Graph Correlation Learning for Disease Diagnosis with Incomplete Labels [66.57101219176275]
  Disease diagnosis on chest X-ray images is a challenging multi-label classification task. We propose a Disease Diagnosis Graph Convolutional Network (DD-GCN) that presents a novel view of investigating the inter-dependency among different diseases. Our method is the first to build a graph over the feature maps with a dynamic adjacency matrix for correlation learning.
  arXiv  Detail & Related papers  (2020-02-26T17:10:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.