Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatment

• URL: http://arxiv.org/abs/2404.12634v3
• Date: Sat, 16 Nov 2024 02:36:32 GMT
• Title: Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatment
• Abstract summary: This study proposes a multi-modal fusion framework Multitrans based on the Transformer architecture and self-attention mechanism. This architecture combines the study of non-contrast computed tomography (NCCT) images and discharge diagnosis reports of patients undergoing stroke treatment.
• Score: 8.686077984641356
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study proposes a multi-modal fusion framework Multitrans based on the Transformer architecture and self-attention mechanism. This architecture combines the study of non-contrast computed tomography (NCCT) images and discharge diagnosis reports of patients undergoing stroke treatment, using a variety of methods based on Transformer architecture approach to predicting functional outcomes of stroke treatment. The results show that the performance of single-modal text classification is significantly better than single-modal image classification, but the effect of multi-modal combination is better than any single modality. Although the Transformer model only performs worse on imaging data, when combined with clinical meta-diagnostic information, both can learn better complementary information and make good contributions to accurately predicting stroke treatment effects..

Related papers

• Dual-Encoder Transformer-Based Multimodal Learning for Ischemic Stroke Lesion Segmentation Using Diffusion MRI [5.332404648315838]
  We study ischemic stroke lesion segmentation using multimodal diffusion MRI from the ISLES 2022 dataset.<n>Several state-of-the-art convolutional and transformer-based architectures, including U-Net variants, Swin-UNet, and TransUNet, are benchmarked.<n>Results show that transformer-based models outperform convolutional baselines, and the proposed dual-encoder TransUNet achieves the best performance, reaching a Dice score of 85.4% on the test set.
  arXiv  Detail & Related papers  (2025-12-23T15:24:31Z)
• Multimodal Medical Endoscopic Image Analysis via Progressive Disentangle-aware Contrastive Learning [11.158864816564538]
  We present an innovative multi-modality representation learning framework based on the Align-Disentangle-Fusion' mechanism.<n>Our method consistently outperforms state-of-the-art approaches, achieving superior accuracy across diverse real clinical scenarios.
  arXiv  Detail & Related papers  (2025-08-23T03:02:51Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• pyMEAL: A Multi-Encoder Augmentation-Aware Learning for Robust and Generalizable Medical Image Translation [0.0]
  3D medical imaging still suffers from data scarcity and inconsistencies due to acquisition protocols, scanner differences, and patient motion.<n>Traditional augmentation uses a single pipeline for all transformations, disregarding the unique traits of each augmentation.<n>We propose a Multi-encoder Augmentation-Aware Learning framework that leverages four distinct augmentation variants processed through dedicated encoders.
  arXiv  Detail & Related papers  (2025-05-30T10:01:23Z)
• Evaluation of Vision Transformers for Multimodal Image Classification: A Case Study on Brain, Lung, and Kidney Tumors [0.0]
  This work evaluates the performance of Vision Transformers architectures, including Swin Transformer and MaxViT, in several datasets. We used three training sets of images with brain, lung, and kidney tumors. Swin Transformer provided high accuracy, achieving up to 99.9% for kidney tumor classification and 99.3% accuracy in a combined dataset.
  arXiv  Detail & Related papers  (2025-02-08T10:35:51Z)
• Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning [5.660131312162423]
  This paper proposes a novel Cross-Graph Modal Contrastive Learning framework for multimodal medical image classification. The proposed approach is evaluated on two datasets: a Parkinson's disease (PD) dataset and a public melanoma dataset. Results demonstrate that CGMCL outperforms conventional unimodal methods in accuracy, interpretability, and early disease prediction.
  arXiv  Detail & Related papers  (2024-10-23T01:25:25Z)
• Translating Imaging to Genomics: Leveraging Transformers for Predictive Modeling [9.403446155541346]
  We aim to bridge the gap between imaging and genomics data by leveraging transformer networks. We propose using only available CT/MRI images to predict genomic sequences.
  arXiv  Detail & Related papers  (2024-08-01T06:14:37Z)
• SeUNet-Trans: A Simple yet Effective UNet-Transformer Model for Medical Image Segmentation [0.0]
  We propose a simple yet effective UNet-Transformer (seUNet-Trans) model for medical image segmentation. In our approach, the UNet model is designed as a feature extractor to generate multiple feature maps from the input images. By leveraging the UNet architecture and the self-attention mechanism, our model not only retains the preservation of both local and global context information but also is capable of capturing long-range dependencies between input elements.
  arXiv  Detail & Related papers  (2023-10-16T01:13:38Z)
• A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics [63.106382317917344]
  We report a Transformer-based representation-learning model as a clinical diagnostic aid that processes multimodal input in a unified manner. The unified model outperformed an image-only model and non-unified multimodal diagnosis models in the identification of pulmonary diseases.
  arXiv  Detail & Related papers  (2023-06-01T16:23:47Z)
• AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images [53.29794593104923]
  We present a novel concept of shared-context processing for whole slide histopathology images. AMIGO uses the celluar graph within the tissue to provide a single representation for a patient. We show that our model is strongly robust to missing information to an extent that it can achieve the same performance with as low as 20% of the data.
  arXiv  Detail & Related papers  (2023-03-01T23:37:45Z)
• TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Prediction [2.358784542343728]
  We propose a transformer-based multimodal network (TranSOP) for a classification approach that employs clinical metadata and imaging information. This includes a fusion module to efficiently combine 3D non-contrast computed tomography (NCCT) features and clinical information. In comparative experiments using unimodal and multimodal data, we achieve a state-of-the-art AUC score of 0.85.
  arXiv  Detail & Related papers  (2023-01-25T21:05:10Z)
• MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer [53.575573940055335]
  We propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2. We verify its effectiveness on 20 medical image segmentation tasks with different image modalities.
  arXiv  Detail & Related papers  (2023-01-19T03:42:36Z)
• CyTran: A Cycle-Consistent Transformer with Multi-Level Consistency for Non-Contrast to Contrast CT Translation [56.622832383316215]
  We propose a novel approach to translate unpaired contrast computed tomography (CT) scans to non-contrast CT scans. Our approach is based on cycle-consistent generative adversarial convolutional transformers, for short, CyTran. Our empirical results show that CyTran outperforms all competing methods.
  arXiv  Detail & Related papers  (2021-10-12T23:25:03Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)
• TransMed: Transformers Advance Multi-modal Medical Image Classification [4.500880052705654]
  convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks. Transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images.
  arXiv  Detail & Related papers  (2021-03-10T08:57:53Z)
• A Multi-Stage Attentive Transfer Learning Framework for Improving COVID-19 Diagnosis [49.3704402041314]
  We propose a multi-stage attentive transfer learning framework for improving COVID-19 diagnosis. Our proposed framework consists of three stages to train accurate diagnosis models through learning knowledge from multiple source tasks and data of different domains. Importantly, we propose a novel self-supervised learning method to learn multi-scale representations for lung CT images.
  arXiv  Detail & Related papers  (2021-01-14T01:39:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.