Translating Imaging to Genomics: Leveraging Transformers for Predictive Modeling

• URL: http://arxiv.org/abs/2408.00311v1
• Date: Thu, 1 Aug 2024 06:14:37 GMT
• Title: Translating Imaging to Genomics: Leveraging Transformers for Predictive Modeling
• Authors: Aiman Farooq, Deepak Mishra, Santanu Chaudhury,
• Abstract summary: We aim to bridge the gap between imaging and genomics data by leveraging transformer networks. We propose using only available CT/MRI images to predict genomic sequences.
• Score: 9.403446155541346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this study, we present a novel approach for predicting genomic information from medical imaging modalities using a transformer-based model. We aim to bridge the gap between imaging and genomics data by leveraging transformer networks, allowing for accurate genomic profile predictions from CT/MRI images. Presently most studies rely on the use of whole slide images (WSI) for the association, which are obtained via invasive methodologies. We propose using only available CT/MRI images to predict genomic sequences. Our transformer based approach is able to efficiently generate associations between multiple sequences based on CT/MRI images alone. This work paves the way for the use of non-invasive imaging modalities for precise and personalized healthcare, allowing for a better understanding of diseases and treatment.

Related papers

• MGI: Multimodal Contrastive pre-training of Genomic and Medical Imaging [16.325123491357203]
  We propose a multimodal pre-training framework that jointly incorporates genomics and medical images for downstream tasks. We align medical images and genes using a self-supervised contrastive learning approach which combines the Mamba as a genetic encoder and the Vision Transformer (ViT) as a medical image encoder.
  arXiv  Detail & Related papers  (2024-06-02T06:20:45Z)
• Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatment [8.686077984641356]
  This study proposes a multi-modal fusion framework Multitrans based on the Transformer architecture and self-attention mechanism. This architecture combines the study of non-contrast computed tomography (NCCT) images and discharge diagnosis reports of patients undergoing stroke treatment.
  arXiv  Detail & Related papers  (2024-04-19T05:31:37Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• Style transfer between Microscopy and Magnetic Resonance Imaging via Generative Adversarial Network in small sample size settings [49.84018914962972]
  Cross-modal augmentation of Magnetic Resonance Imaging (MRI) and microscopic imaging based on the same tissue samples is promising. We tested a method for generating microscopic histological images from MRI scans of the corpus callosum using conditional generative adversarial network (cGAN) architecture.
  arXiv  Detail & Related papers  (2023-10-16T13:58:53Z)
• On Sensitivity and Robustness of Normalization Schemes to Input Distribution Shifts in Automatic MR Image Diagnosis [58.634791552376235]
  Deep Learning (DL) models have achieved state-of-the-art performance in diagnosing multiple diseases using reconstructed images as input. DL models are sensitive to varying artifacts as it leads to changes in the input data distribution between the training and testing phases. We propose to use other normalization techniques, such as Group Normalization and Layer Normalization, to inject robustness into model performance against varying image artifacts.
  arXiv  Detail & Related papers  (2023-06-23T03:09:03Z)
• Conversion Between CT and MRI Images Using Diffusion and Score-Matching Models [7.745729132928934]
  We propose to use an emerging deep learning framework called diffusion and score-matching models. Our results show that the diffusion and score-matching models generate better synthetic CT images than the CNN and GAN models. Our study suggests that diffusion and score-matching models are powerful to generate high quality images conditioned on an image obtained using a complementary imaging modality.
  arXiv  Detail & Related papers  (2022-09-24T23:50:54Z)
• Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation [75.58395328700821]
  We propose a novel model, Multi-modal Gaussian Process Prior Variational Autoencoder (MGP-VAE), to impute one or more missing sub-modalities for a patient scan. MGP-VAE can leverage the Gaussian Process (GP) prior on the Variational Autoencoder (VAE) to utilize the subjects/patients and sub-modalities correlations. We show the applicability of MGP-VAE on brain tumor segmentation where either, two, or three of four sub-modalities may be missing.
  arXiv  Detail & Related papers  (2021-07-07T19:06:34Z)
• Learning stochastic object models from medical imaging measurements by use of advanced AmbientGANs [7.987904193401004]
  generative adversarial networks (GANs) hold potential for such tasks. Deep generative neural networks, such as generative adversarial networks (GANs) hold potential for such tasks. In this work, a modified AmbientGAN training strategy is proposed that is suitable for modern progressive or multi-resolution training approaches.
  arXiv  Detail & Related papers  (2021-06-27T21:46:23Z)
• SAG-GAN: Semi-Supervised Attention-Guided GANs for Data Augmentation on Medical Images [47.35184075381965]
  We present a data augmentation method for generating synthetic medical images using cycle-consistency Generative Adversarial Networks (GANs) The proposed GANs-based model can generate a tumor image from a normal image, and in turn, it can also generate a normal image from a tumor image. We train the classification model using real images with classic data augmentation methods and classification models using synthetic images.
  arXiv  Detail & Related papers  (2020-11-15T14:01:24Z)
• Structurally aware bidirectional unpaired image to image translation between CT and MR [0.14788776577018314]
  Deep learning techniques can help us to leverage the possibility of an image to image translation between multiple imaging modalities. These techniques will help to conduct surgical planning under CT with the feedback of MRI information.
  arXiv  Detail & Related papers  (2020-06-05T11:21:56Z)
• Pathological Retinal Region Segmentation From OCT Images Using Geometric Relation Based Augmentation [84.7571086566595]
  We propose improvements over previous GAN-based medical image synthesis methods by jointly encoding the intrinsic relationship of geometry and shape. The proposed method outperforms state-of-the-art segmentation methods on the public RETOUCH dataset having images captured from different acquisition procedures.
  arXiv  Detail & Related papers  (2020-03-31T11:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.