An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion

• URL: http://arxiv.org/abs/2606.20738v1
• Date: Wed, 17 Jun 2026 16:43:01 GMT
• Title: An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion
• Abstract summary: This article presents a complementary approach for integrating multimodal medical data in cancer classification.<n>A mixed multimodal fusion architecture, called Mixed Fusion, was employed to enhance the interpretability of the decision-making process.
• Score: 1.5420873135976756
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This article presents a complementary approach for integrating multimodal medical data in cancer classification, based on state space models represented by the Mamba architecture. To this end, a mixed multimodal fusion architecture, called Mixed Fusion, was employed and developed to enhance the interpretability of the decision-making process. The proposed approach explores two variants of Mamba: one dedicated to visual processing, responsible for classifying the lesion image and generating probabilities associated with the target classes, and another focused on tabular processing, which uses these probabilities together with clinical and/or sociodemographic data to produce the final diagnosis. The experiments were conducted on two medical datasets: PAD-UFES-20, composed of clinical images and information associated with skin lesions, and NDB-UFES, consisting of histopathological images and sociodemographic data related to oral cancer. The results indicate slightly lower performance in balanced accuracy, compared with Transformer-based approaches, on PAD-UFES-20, and superior performance on NDB-UFES. Additionally, substantial gains were observed in the recall metric. Furthermore, the adoption of the Mixed Fusion architecture enables the application of the Shapley Additive Explanations (SHAP) method, increasing the interpretability of the results. These findings indicate that Mamba-based models constitute a suitable alternative for multimodal classification in medical data, especially in scenarios in which sensitivity is a relevant requirement.

Related papers

• Dementia Etiology Diagnosis via Collaborative Meta Knowledge Enhancement [81.27196033392559]
  We propose a Collaborative Meta Knowledge Enhancement (COME) framework for dementia etiology diagnosis.<n>COME injects multi-center acquisition semantics, source identifiers, and modality indicators as heterogeneity-aware embeddings into a unified Transformer architecture for scale-up training.<n>Our method achieves state-of-the-art in-domain performance with a mean macro-averaged AUC of 85.62% and a 4.29-point gain over the strongest baseline.
  arXiv  Detail & Related papers  (2026-07-24T04:26:47Z)
• Multimodal Graph-based Classification of Esophageal Motility Disorders [73.90451172929117]
  Diagnosing esophageal motility disorders pose significant challenges due to the complexity of high-resolution impedance manometry data and variability in clinical interpretation.<n>This work explores the feasibility of a multimodal Machine Learning-based classification approach that combines HRIM recordings with patient-specific information and incorporates a graph-based modeling of esophageal physiology.
  arXiv  Detail & Related papers  (2026-05-13T14:52:12Z)
• Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention [1.407401787551479]
  We propose a multimodal latent diffusion model that jointly synthesizes magnetic resonance imaging (MRI) and tabular clinical data.<n>Our model utilizes a variational autoencoder to fuse the two modalities before diffusion-based synthesis.<n>We evaluated the framework on data from the German National Cohort (NAKO Gesundheitsstudie)
  arXiv  Detail & Related papers  (2026-05-05T08:30:20Z)
• GraphMMP: A Graph Neural Network Model with Mutual Information and Global Fusion for Multimodal Medical Prognosis [11.256328387342252]
  This paper presents a two-stage multimodal prognosis model, GraphMMP, based on graph neural networks.<n>The proposed model constructs feature graphs using mutual information and features a global fusion module built on Mamba, which significantly boosts prognosis performance.<n> Empirical results show that GraphMMP surpasses existing methods on datasets related to liver prognosis and the METABRIC study, demonstrating its effectiveness in multimodal medical prognosis tasks.
  arXiv  Detail & Related papers  (2025-08-24T18:06:20Z)
• Multi-Omics Fusion with Soft Labeling for Enhanced Prediction of Distant Metastasis in Nasopharyngeal Carcinoma Patients after Radiotherapy [4.971538849792411]
  One of the challenges encountered in the integration of omics data is the presence of unpredictability.<n>This study aims to develop a fusion methodology that mitigates the disparities inherent in omics data.
  arXiv  Detail & Related papers  (2025-02-12T05:26:59Z)
• Multimodal Medical Disease Classification with LLaMA II [0.14999444543328289]
  We use the text-image pair dataset from OpenI consisting of 2D chest X-rays associated with clinical reports.<n>Our focus is on fusion methods for merging text and vision information extracted from medical datasets.<n>The newly introduced multimodal architecture can be applied to other multimodal datasets with little effort and can be easily adapted for further research.
  arXiv  Detail & Related papers  (2024-12-02T09:18:07Z)
• KAN-Mamba FusionNet: Redefining Medical Image Segmentation with Non-Linear Modeling [3.2971993272923443]
  We propose a novel architecture, the KAN-Mamba FusionNet, to improve medical image segmentation accuracy.<n>It consistently outperforms state-of-the-art methods in IoU and F1 scores.
  arXiv  Detail & Related papers  (2024-11-18T09:19:16Z)
• XAI for In-hospital Mortality Prediction via Multimodal ICU Data [57.73357047856416]
  We propose an efficient, explainable AI solution for predicting in-hospital mortality via multimodal ICU data. We employ multimodal learning in our framework, which can receive heterogeneous inputs from clinical data and make decisions. Our framework can be easily transferred to other clinical tasks, which facilitates the discovery of crucial factors in healthcare research.
  arXiv  Detail & Related papers  (2023-12-29T14:28:04Z)
• MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer [53.575573940055335]
  We propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2. We verify its effectiveness on 20 medical image segmentation tasks with different image modalities.
  arXiv  Detail & Related papers  (2023-01-19T03:42:36Z)
• MMLN: Leveraging Domain Knowledge for Multimodal Diagnosis [10.133715767542386]
  We propose a knowledge-driven and data-driven framework for lung disease diagnosis. We formulate diagnosis rules according to authoritative clinical medicine guidelines and learn the weights of rules from text data. A multimodal fusion consisting of text and image data is designed to infer the marginal probability of lung disease.
  arXiv  Detail & Related papers  (2022-02-09T04:12:30Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.