Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention

• URL: http://arxiv.org/abs/2605.06699v1
• Date: Tue, 05 May 2026 08:30:20 GMT
• Title: Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention
• Abstract summary: We propose a multimodal latent diffusion model that jointly synthesizes magnetic resonance imaging (MRI) and tabular clinical data.<n>Our model utilizes a variational autoencoder to fuse the two modalities before diffusion-based synthesis.<n>We evaluated the framework on data from the German National Cohort (NAKO Gesundheitsstudie)
• Score: 1.407401787551479
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a multimodal latent diffusion model that jointly synthesizes volumetric magnetic resonance imaging (MRI) and tabular clinical data within a shared latent space via cross-attention. This approach enables coherent joint representation learning of MRI and tabular modalities for generative modeling. Our model utilizes a variational autoencoder to fuse the two modalities before diffusion-based synthesis, allowing modality-appropriate reconstruction with separate decoders for MRI and tabular data. We evaluated the framework on data from the German National Cohort (NAKO Gesundheitsstudie), comprising over 10,000 participants with MRI scans and clinical tabular features such as age, sex, body measurements, and ethnicity. The generated MRI volumes exhibited anatomical plausibility and body composition consistent with the synthesized tabular attributes. Quantitative evaluation using Fréchet distance and precision-recall metrics confirmed high-fidelity image generation. In the tabular modality, our model outperformed CTGAN across standard evaluation metrics and achieved results comparable to TVAE, demonstrating competitive performance relative to established unimodal baselines. This work is, to our knowledge, the first to demonstrate the feasibility of jointly modeling MRI and mixed-type tabular data in a single latent diffusion framework, offering a proof-of-concept for generating coherent synthetic multimodal patient data and aligning with the broader goal of developing digital twins in healthcare.

Related papers

• Wavelet-Fusion Diffusion Model for Multimodal Brain MRI Synthesis with Modality and Metadata Conditioning [0.0]
  Multimodal MRI provides complementary information for neuroimaging analysis, where different imaging modalities capture distinct anatomical, tissue, and pathological features.<n>Large-scale structural MRI resources are increasingly available, but their modality coverage is often uneven across public and pooled neuroimaging datasets.<n>Our approach combines a Wavelet-Fusion variational autoencoder (WF-VAE) latent compressor with a conditional 3D U-Net diffusion model trained in the learned latent space.
  arXiv  Detail & Related papers  (2026-05-30T11:57:27Z)
• Mamba-driven MRI-to-CT Synthesis for MRI-only Radiotherapy Planning [4.3531418255555545]
  Mamba is a state-space model for cross-modality translation.<n>We adapt both the U-Mamba and the SegMamba architecture, originally proposed for segmentation, to perform cross-modality image generation.<n>Our 3D Mamba architecture effectively captures complex features and long-range volumetric dependencies, thus allowing accurate CT synthesis.
  arXiv  Detail & Related papers  (2026-03-24T14:56:39Z)
• A Unified Attention U-Net Framework for Cross-Modality Tumor Segmentation in MRI and CT [0.0]
  This study presents a unified Attention U-Net architecture trained jointly on MRI (BraTS 2021) and CT (LIDC-IDRI) datasets.<n>Our proposed pipeline incorporates modality-harmonized preprocessing, attention-gated skip connections, and a modality-aware Focal Tversky loss function.
  arXiv  Detail & Related papers  (2026-01-07T23:50:45Z)
• Diffusion Models for conditional MRI generation [0.0]
  We present a Latent Diffusion Model (LDM) for the generation of brain Magnetic Resonance Imaging (MRI)<n>To evaluate the quality of the generated images, the Fr'echet Inception Distance (FID) and Multi-Scale Structural Similarity Index (MS-SSIM) metrics were employed.
  arXiv  Detail & Related papers  (2025-02-25T20:08:29Z)
• MRGen: Segmentation Data Engine for Underrepresented MRI Modalities [59.61465292965639]
  Training medical image segmentation models for rare yet clinically important imaging modalities is challenging due to the scarcity of annotated data.<n>This paper investigates leveraging generative models to synthesize data, for training segmentation models for underrepresented modalities.<n>We present MRGen, a data engine for controllable medical image synthesis conditioned on text prompts and segmentation masks.
  arXiv  Detail & Related papers  (2024-12-04T16:34:22Z)
• Physics-Informed Latent Diffusion for Multimodal Brain MRI Synthesis [43.82741134285203]
  We present a physics-informed generative model capable of synthesizing a variable number of brain MRI modalities. Our approach utilizes latent diffusion models and a two-step generative process. Experiments demonstrate the efficacy of this approach in generating unseen MR contrasts and preserving physical plausibility.
  arXiv  Detail & Related papers  (2024-09-20T14:21:34Z)
• Principled Feature Disentanglement for High-Fidelity Unified Brain MRI Synthesis [5.657045602016982]
  We propose a novel unified framework for multisequence MR images, called hybrid-fusion GAN (HF-GAN)<n>A powerful many-to-one stream is constructed for the extraction of complex complementary features, while utilizing parallel, one-to-one streams to process modality-specific information.<n>We validated our framework on public datasets of both healthy and pathological brain MRI.
  arXiv  Detail & Related papers  (2024-06-21T08:06:00Z)
• HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling [4.44283662576491]
  We present a novel framework based on hypernetworks to fuse clinical imaging and tabular data by conditioning the image processing on the EHR's values and measurements.<n>This approach aims to leverage the complementary information present in these modalities to enhance the accuracy of various medical applications.
  arXiv  Detail & Related papers  (2024-03-20T05:50:04Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading [47.50733518140625]
  Brain tumor represents one of the most fatal cancers around the world, and is very common in children and the elderly. We propose a novel cross-modality guidance-aided multi-modal learning with dual attention for addressing the task of MRI brain tumor grading.
  arXiv  Detail & Related papers  (2024-01-17T07:54:49Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)
• Confidence-guided Lesion Mask-based Simultaneous Synthesis of Anatomic and Molecular MR Images in Patients with Post-treatment Malignant Gliomas [65.64363834322333]
  Confidence Guided SAMR (CG-SAMR) synthesizes data from lesion information to multi-modal anatomic sequences. module guides the synthesis based on confidence measure about the intermediate results. experiments on real clinical data demonstrate that the proposed model can perform better than the state-of-theart synthesis methods.
  arXiv  Detail & Related papers  (2020-08-06T20:20:22Z)
• Lesion Mask-based Simultaneous Synthesis of Anatomic and MolecularMR Images using a GAN [59.60954255038335]
  The proposed framework consists of a stretch-out up-sampling module, a brain atlas encoder, a segmentation consistency module, and multi-scale label-wise discriminators. Experiments on real clinical data demonstrate that the proposed model can perform significantly better than the state-of-the-art synthesis methods.
  arXiv  Detail & Related papers  (2020-06-26T02:50:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.