ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics

• URL: http://arxiv.org/abs/2111.13424v1
• Date: Fri, 26 Nov 2021 11:06:12 GMT
• Title: ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics
• Authors: Aiham Taleb, Matthias Kirchler, Remo Monti, Christoph Lippert
• Abstract summary: We propose ContIG, a self-supervised method that can learn from large datasets of unlabeled medical images and genetic data. Our approach aligns images and several genetic modalities in the feature space using a contrastive loss. We also perform genome-wide association studies on the features learned by our models, uncovering interesting relationships between images and genetic data.
• Score: 4.907551775445731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High annotation costs are a substantial bottleneck in applying modern deep learning architectures to clinically relevant medical use cases, substantiating the need for novel algorithms to learn from unlabeled data. In this work, we propose ContIG, a self-supervised method that can learn from large datasets of unlabeled medical images and genetic data. Our approach aligns images and several genetic modalities in the feature space using a contrastive loss. We design our method to integrate multiple modalities of each individual person in the same model end-to-end, even when the available modalities vary across individuals. Our procedure outperforms state-of-the-art self-supervised methods on all evaluated downstream benchmark tasks. We also adapt gradient-based explainability algorithms to better understand the learned cross-modal associations between the images and genetic modalities. Finally, we perform genome-wide association studies on the features learned by our models, uncovering interesting relationships between images and genetic data.

Related papers

• Coupling AI and Citizen Science in Creation of Enhanced Training Dataset for Medical Image Segmentation [3.7274206780843477]
  We introduce a robust and versatile framework that combines AI and crowdsourcing to improve the quality and quantity of medical image datasets. Our approach utilise a user-friendly online platform that enables a diverse group of crowd annotators to label medical images efficiently. We employ pix2pixGAN, a generative AI model, to expand the training dataset with synthetic images that capture realistic morphological features.
  arXiv  Detail & Related papers  (2024-09-04T21:22:54Z)
• MGI: Multimodal Contrastive pre-training of Genomic and Medical Imaging [16.325123491357203]
  We propose a multimodal pre-training framework that jointly incorporates genomics and medical images for downstream tasks. We align medical images and genes using a self-supervised contrastive learning approach which combines the Mamba as a genetic encoder and the Vision Transformer (ViT) as a medical image encoder.
  arXiv  Detail & Related papers  (2024-06-02T06:20:45Z)
• Genetic InfoMax: Exploring Mutual Information Maximization in High-Dimensional Imaging Genetics Studies [50.11449968854487]
  Genome-wide association studies (GWAS) are used to identify relationships between genetic variations and specific traits. Representation learning for imaging genetics is largely under-explored due to the unique challenges posed by GWAS. We introduce a trans-modal learning framework Genetic InfoMax (GIM) to address the specific challenges of GWAS.
  arXiv  Detail & Related papers  (2023-09-26T03:59:21Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• Incomplete Multimodal Learning for Complex Brain Disorders Prediction [65.95783479249745]
  We propose a new incomplete multimodal data integration approach that employs transformers and generative adversarial networks. We apply our new method to predict cognitive degeneration and disease outcomes using the multimodal imaging genetic data from Alzheimer's Disease Neuroimaging Initiative cohort.
  arXiv  Detail & Related papers  (2023-05-25T16:29:16Z)
• Domain Generalization for Mammographic Image Analysis with Contrastive Learning [62.25104935889111]
  The training of an efficacious deep learning model requires large data with diverse styles and qualities. A novel contrastive learning is developed to equip the deep learning models with better style generalization capability. The proposed method has been evaluated extensively and rigorously with mammograms from various vendor style domains and several public datasets.
  arXiv  Detail & Related papers  (2023-04-20T11:40:21Z)
• AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images [53.29794593104923]
  We present a novel concept of shared-context processing for whole slide histopathology images. AMIGO uses the celluar graph within the tissue to provide a single representation for a patient. We show that our model is strongly robust to missing information to an extent that it can achieve the same performance with as low as 20% of the data.
  arXiv  Detail & Related papers  (2023-03-01T23:37:45Z)
• Metadata-enhanced contrastive learning from retinal optical coherence tomography images [7.932410831191909]
  We extend conventional contrastive frameworks with a novel metadata-enhanced strategy. Our approach employs widely available patient metadata to approximate the true set of inter-image contrastive relationships. Our approach outperforms both standard contrastive methods and a retinal image foundation model in five out of six image-level downstream tasks.
  arXiv  Detail & Related papers  (2022-08-04T08:53:15Z)
• Federated Learning for Computational Pathology on Gigapixel Whole Slide Images [4.035591045544291]
  We introduce privacy-preserving federated learning for gigapixel whole slide images in computational pathology. We evaluate our approach on two different diagnostic problems using thousands of histology whole slide images with only slide-level labels.
  arXiv  Detail & Related papers  (2020-09-21T21:56:08Z)
• Select-ProtoNet: Learning to Select for Few-Shot Disease Subtype Prediction [55.94378672172967]
  We focus on few-shot disease subtype prediction problem, identifying subgroups of similar patients. We introduce meta learning techniques to develop a new model, which can extract the common experience or knowledge from interrelated clinical tasks. Our new model is built upon a carefully designed meta-learner, called Prototypical Network, that is a simple yet effective meta learning machine for few-shot image classification.
  arXiv  Detail & Related papers  (2020-09-02T02:50:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.