MEDFORM: A Foundation Model for Contrastive Learning of CT Imaging and Clinical Numeric Data in Multi-Cancer Analysis

• URL: http://arxiv.org/abs/2501.13277v1
• Date: Wed, 22 Jan 2025 23:56:37 GMT
• Title: MEDFORM: A Foundation Model for Contrastive Learning of CT Imaging and Clinical Numeric Data in Multi-Cancer Analysis
• Authors: Daeun Jung, Jaehyeok Jang, Sooyoung Jang, Yu Rang Park,
• Abstract summary: We propose MEDFORM, a multimodal pre-training strategy that guides CT image representation learning. MEDFORM efficiently processes CT slice through multiple instance learning (MIL) and adopts a dual pre-training strategy. Our model was pre-trained on three different cancer types: lung cancer (141,171 slices), breast cancer (8,100 slices), colorectal cancer (10,393 slices)
• Score: 0.562479170374811
• License:
• Abstract: Computed tomography (CT) and clinical numeric data are essential modalities for cancer evaluation, but building large-scale multimodal training datasets for developing medical foundation models remains challenging due to the structural complexity of multi-slice CT data and high cost of expert annotation. In this study, we propose MEDFORM, a multimodal pre-training strategy that guides CT image representation learning using complementary information from clinical data for medical foundation model development. MEDFORM efficiently processes CT slice through multiple instance learning (MIL) and adopts a dual pre-training strategy: first pretraining the CT slice feature extractor using SimCLR-based self-supervised learning, then aligning CT and clinical modalities through cross-modal contrastive learning. Our model was pre-trained on three different cancer types: lung cancer (141,171 slices), breast cancer (8,100 slices), colorectal cancer (10,393 slices). The experimental results demonstrated that this dual pre-training strategy improves cancer classification performance and maintains robust performance in few-shot learning scenarios. Code available at https://github.com/DigitalHealthcareLab/25MultiModalFoundationModel.git

Related papers

• CC-DCNet: Dynamic Convolutional Neural Network with Contrastive Constraints for Identifying Lung Cancer Subtypes on Multi-modality Images [13.655407979403945]
  We propose a novel deep learning network designed to accurately classify lung cancer subtype with multi-dimensional and multi-modality images. The strength of the proposed model lies in its ability to dynamically process both paired CT-pathological image sets and independent CT image sets. We also develop a contrastive constraint module, which quantitatively maps the cross-modality associations through network training.
  arXiv  Detail & Related papers  (2024-07-18T01:42:00Z)
• A Unified Multi-Phase CT Synthesis and Classification Framework for Kidney Cancer Diagnosis with Incomplete Data [18.15801599933636]
  We propose a unified framework for kidney cancer diagnosis with incomplete multi-phase CT. It simultaneously recovers missing CT images and classifies cancer subtypes using the completed set of images. The proposed framework is based on fully 3D convolutional neural networks.
  arXiv  Detail & Related papers  (2023-12-09T11:34:14Z)
• MUSCLE: Multi-task Self-supervised Continual Learning to Pre-train Deep Models for X-ray Images of Multiple Body Parts [63.30352394004674]
  Multi-task Self-super-vised Continual Learning (MUSCLE) is a novel self-supervised pre-training pipeline for medical imaging tasks. MUSCLE aggregates X-rays collected from multiple body parts for representation learning, and adopts a well-designed continual learning procedure. We evaluate MUSCLE using 9 real-world X-ray datasets with various tasks, including pneumonia classification, skeletal abnormality classification, lung segmentation, and tuberculosis (TB) detection.
  arXiv  Detail & Related papers  (2023-10-03T12:19:19Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics [63.106382317917344]
  We report a Transformer-based representation-learning model as a clinical diagnostic aid that processes multimodal input in a unified manner. The unified model outperformed an image-only model and non-unified multimodal diagnosis models in the identification of pulmonary diseases.
  arXiv  Detail & Related papers  (2023-06-01T16:23:47Z)
• Competence-based Multimodal Curriculum Learning for Medical Report Generation [98.10763792453925]
  We propose a Competence-based Multimodal Curriculum Learning framework ( CMCL) to alleviate the data bias and make best use of available data. Specifically, CMCL simulates the learning process of radiologists and optimize the model in a step by step manner. Experiments on the public IU-Xray and MIMIC-CXR datasets show that CMCL can be incorporated into existing models to improve their performance.
  arXiv  Detail & Related papers  (2022-06-24T08:16:01Z)
• Cross-Modality Deep Feature Learning for Brain Tumor Segmentation [158.8192041981564]
  This paper proposes a novel cross-modality deep feature learning framework to segment brain tumors from the multi-modality MRI data. The core idea is to mine rich patterns across the multi-modality data to make up for the insufficient data scale. Comprehensive experiments are conducted on the BraTS benchmarks, which show that the proposed cross-modality deep feature learning framework can effectively improve the brain tumor segmentation performance.
  arXiv  Detail & Related papers  (2022-01-07T07:46:01Z)
• Segmentation of Lung Tumor from CT Images using Deep Supervision [0.8733639720576208]
  Lung cancer is a leading cause of death in most countries of the world. This paper approaches lung tumor segmentation by applying two-dimensional discrete wavelet transform (DWT) on the LOTUS dataset.
  arXiv  Detail & Related papers  (2021-11-17T17:50:18Z)
• CT-SGAN: Computed Tomography Synthesis GAN [4.765541373485143]
  We propose the CT-SGAN model that generates large-scale 3D synthetic CT-scan volumes when trained on a small dataset of chest CT-scans. We show that CT-SGAN can significantly improve lung detection accuracy by pre-training a nodule on a vast amount of synthetic data.
  arXiv  Detail & Related papers  (2021-10-14T22:20:40Z)
• A Multi-Stage Attentive Transfer Learning Framework for Improving COVID-19 Diagnosis [49.3704402041314]
  We propose a multi-stage attentive transfer learning framework for improving COVID-19 diagnosis. Our proposed framework consists of three stages to train accurate diagnosis models through learning knowledge from multiple source tasks and data of different domains. Importantly, we propose a novel self-supervised learning method to learn multi-scale representations for lung CT images.
  arXiv  Detail & Related papers  (2021-01-14T01:39:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.