Multi-stage intermediate fusion for multimodal learning to classify non-small cell lung cancer subtypes from CT and PET

• URL: http://arxiv.org/abs/2501.12425v1
• Date: Tue, 21 Jan 2025 12:10:00 GMT
• Title: Multi-stage intermediate fusion for multimodal learning to classify non-small cell lung cancer subtypes from CT and PET
• Authors: Fatih Aksu, Fabrizia Gelardi, Arturo Chiti, Paolo Soda,
• Abstract summary: This study presents a multi-stage intermediate fusion approach to classify NSCLC subtypes from CT and PET images. Our method integrates the two modalities at different stages of feature extraction, using voxel-wise fusion to exploit complementary information. Our results demonstrate that the proposed method outperforms all alternatives across key metrics, with an accuracy and AUC equal to 0.724 and 0.681, respectively.
• Score: 0.43498389175652047
• License:
• Abstract: Accurate classification of histological subtypes of non-small cell lung cancer (NSCLC) is essential in the era of precision medicine, yet current invasive techniques are not always feasible and may lead to clinical complications. This study presents a multi-stage intermediate fusion approach to classify NSCLC subtypes from CT and PET images. Our method integrates the two modalities at different stages of feature extraction, using voxel-wise fusion to exploit complementary information across varying abstraction levels while preserving spatial correlations. We compare our method against unimodal approaches using only CT or PET images to demonstrate the benefits of modality fusion, and further benchmark it against early and late fusion techniques to highlight the advantages of intermediate fusion during feature extraction. Additionally, we compare our model with the only existing intermediate fusion method for histological subtype classification using PET/CT images. Our results demonstrate that the proposed method outperforms all alternatives across key metrics, with an accuracy and AUC equal to 0.724 and 0.681, respectively. This non-invasive approach has the potential to significantly improve diagnostic accuracy, facilitate more informed treatment decisions, and advance personalized care in lung cancer management.

Related papers

• Multi-modal Medical Image Fusion For Non-Small Cell Lung Cancer Classification [7.002657345547741]
  Non-small cell lung cancer (NSCLC) is a predominant cause of cancer mortality worldwide. In this paper, we introduce an innovative integration of multi-modal data, synthesizing fused medical imaging (CT and PET scans) with clinical health records and genomic data. Our research surpasses existing approaches, as evidenced by a substantial enhancement in NSCLC detection and classification precision.
  arXiv  Detail & Related papers  (2024-09-27T12:59:29Z)
• Let it shine: Autofluorescence of Papanicolaou-stain improves AI-based cytological oral cancer detection [3.1850395068284785]
  Oral cancer is treatable if detected early, but it is often fatal in late stages. Computer-assisted methods are essential for cost-effective and accurate cytological analysis. This study aims to improve AI-based oral cancer detection using multimodal imaging and deep fusion.
  arXiv  Detail & Related papers  (2024-07-02T01:05:35Z)
• PEMMA: Parameter-Efficient Multi-Modal Adaptation for Medical Image Segmentation [5.056996354878645]
  When both CT and PET scans are available, it is common to combine them as two channels of the input to the segmentation model. This method requires both scan types during training and inference, posing a challenge due to the limited availability of PET scans. We propose a parameter-efficient multi-modal adaptation framework for lightweight upgrading of a transformer-based segmentation model.
  arXiv  Detail & Related papers  (2024-04-21T16:29:49Z)
• Contrastive Diffusion Model with Auxiliary Guidance for Coarse-to-Fine PET Reconstruction [62.29541106695824]
  This paper presents a coarse-to-fine PET reconstruction framework that consists of a coarse prediction module (CPM) and an iterative refinement module (IRM) By delegating most of the computational overhead to the CPM, the overall sampling speed of our method can be significantly improved. Two additional strategies, i.e., an auxiliary guidance strategy and a contrastive diffusion strategy, are proposed and integrated into the reconstruction process.
  arXiv  Detail & Related papers  (2023-08-20T04:10:36Z)
• Classification of lung cancer subtypes on CT images with synthetic pathological priors [41.75054301525535]
  Cross-scale associations exist in the image patterns between the same case's CT images and its pathological images. We propose self-generating hybrid feature network (SGHF-Net) for accurately classifying lung cancer subtypes on CT images.
  arXiv  Detail & Related papers  (2023-08-09T02:04:05Z)
• Improving Classification Model Performance on Chest X-Rays through Lung Segmentation [63.45024974079371]
  We propose a deep learning approach to enhance abnormal chest x-ray (CXR) identification performance through segmentations. Our approach is designed in a cascaded manner and incorporates two modules: a deep neural network with criss-cross attention modules (XLSor) for localizing lung region in CXR images and a CXR classification model with a backbone of a self-supervised momentum contrast (MoCo) model pre-trained on large-scale CXR data sets.
  arXiv  Detail & Related papers  (2022-02-22T15:24:06Z)
• Multiple Time Series Fusion Based on LSTM An Application to CAP A Phase Classification Using EEG [56.155331323304]
  Deep learning based electroencephalogram channels' feature level fusion is carried out in this work. Channel selection, fusion, and classification procedures were optimized by two optimization algorithms.
  arXiv  Detail & Related papers  (2021-12-18T14:17:49Z)
• A Multi-Stage Attentive Transfer Learning Framework for Improving COVID-19 Diagnosis [49.3704402041314]
  We propose a multi-stage attentive transfer learning framework for improving COVID-19 diagnosis. Our proposed framework consists of three stages to train accurate diagnosis models through learning knowledge from multiple source tasks and data of different domains. Importantly, we propose a novel self-supervised learning method to learn multi-scale representations for lung CT images.
  arXiv  Detail & Related papers  (2021-01-14T01:39:19Z)
• Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation [11.622615048002567]
  Multimodal spatial attention module (MSAM) learns to emphasize regions related to tumors. MSAM can be applied to common backbone architectures and trained end-to-end.
  arXiv  Detail & Related papers  (2020-07-29T10:27:22Z)
• Co-Heterogeneous and Adaptive Segmentation from Multi-Source and Multi-Phase CT Imaging Data: A Study on Pathological Liver and Lesion Segmentation [48.504790189796836]
  We present a novel segmentation strategy, co-heterogenous and adaptive segmentation (CHASe) We propose a versatile framework that fuses appearance based semi-supervision, mask based adversarial domain adaptation, and pseudo-labeling. CHASe can further improve pathological liver mask Dice-Sorensen coefficients by ranges of $4.2% sim 9.4%$.
  arXiv  Detail & Related papers  (2020-05-27T06:58:39Z)
• Detecting Pancreatic Ductal Adenocarcinoma in Multi-phase CT Scans via Alignment Ensemble [77.5625174267105]
  Pancreatic ductal adenocarcinoma (PDAC) is one of the most lethal cancers among the population. Multiple phases provide more information than single phase, but they are unaligned and inhomogeneous in texture. We suggest an ensemble of all these alignments as a promising way to boost the performance of PDAC detection.
  arXiv  Detail & Related papers  (2020-03-18T19:06:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.