A Generative Foundation Model for Multimodal Histopathology

• URL: http://arxiv.org/abs/2604.03635v1
• Date: Sat, 04 Apr 2026 08:09:11 GMT
• Title: A Generative Foundation Model for Multimodal Histopathology
• Abstract summary: We introduce MuPD, a generative foundation model that embeds hematoxylin and eosin (H&E)-stained histology, molecular RNA profiles, and clinical text into a shared latent space.<n>MuPD supports diverse cross-modal tasks with minimal or no task-specific fine-tuning.<n>For text-conditioned and image-to-image generation, MuPD synthesizes histologically faithful tissue architectures.
• Score: 25.341102846996453
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate diagnosis and treatment of complex diseases require integrating histological, molecular, and clinical data, yet in practice these modalities are often incomplete owing to tissue scarcity, assay cost, and workflow constraints. Existing computational approaches attempt to impute missing modalities from available data but rely on task-specific models trained on narrow, single source-target pairs, limiting their generalizability. Here we introduce MuPD (Multimodal Pathology Diffusion), a generative foundation model that embeds hematoxylin and eosin (H&E)-stained histology, molecular RNA profiles, and clinical text into a shared latent space through a diffusion transformer with decoupled cross-modal attention. Pretrained on 100 million histology image patches, 1.6 million text-histology pairs, and 10.8 million RNA-histology pairs spanning 34 human organs, MuPD supports diverse cross-modal synthesis tasks with minimal or no task-specific fine-tuning. For text-conditioned and image-to-image generation, MuPD synthesizes histologically faithful tissue architectures, reducing Fréchet inception distance (FID) scores by 50% relative to domain-specific models and improving few-shot classification accuracy by up to 47% through synthetic data augmentation. For RNA-conditioned histology generation, MuPD reduces FID by 23% compared with the next-best method while preserving cell-type distributions across five cancer types. As a virtual stainer, MuPD translates H&E images to immunohistochemistry and multiplex immunofluorescence, improving average marker correlation by 37% over existing approaches. These results demonstrate that a single, unified generative model pretrained across heterogeneous pathology modalities can substantially outperform specialized alternatives, providing a scalable computational framework for multimodal histopathology.

Related papers

• Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction [3.229566379058554]
  We show that training a lightweight alignment module atop frozen histopathology and RNA-Seq foundation models enables open-vocabulary molecular prompting.<n>We achieve a 25-fold improvement in retrieval over baseline methods.<n>We validate clinical utility on the POSEIDON clinical trial.
  arXiv  Detail & Related papers  (2026-06-29T08:24:53Z)
• Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology [47.79180201495836]
  MixTIME is a multimodal foundation model that integrates pathology foundation models trained across distinct modalities.<n>It predicts multiplex immunofluorescence (mIF) protein expression from hematoxylin and eosin whole-slide images.<n>MixTIME achieves state-of-the-art performance across 17 protein markers as measured by correlation metrics.
  arXiv  Detail & Related papers  (2026-06-16T16:22:42Z)
• PRIME: Prototype-Driven Multimodal Pretraining for Cancer Prognosis with Missing Modalities [86.63247982275396]
  PRIME is a missing-aware multimodal self-supervised pretraining framework.<n>It learns robust and transferable representations from partially observed cohorts.<n>We evaluate PRIME on The Cancer Genome Atlas with label-free pretraining on 32 cancer types.
  arXiv  Detail & Related papers  (2026-04-05T21:14:27Z)
• R-GenIMA: Integrating Neuroimaging and Genetics with Interpretable Multimodal AI for Alzheimer's Disease Progression [63.97617759805451]
  Early detection of Alzheimer's disease requires models capable of integrating macro-scale neuroanatomical alterations with micro-scale genetic susceptibility.<n>We introduce R-GenIMA, an interpretable multimodal large language model that couples a novel ROI-wise vision transformer with genetic prompting.<n>R-GenIMA achieves state-of-the-art performance in four-way classification across normal cognition, subjective memory concerns, mild cognitive impairment, and AD.
  arXiv  Detail & Related papers  (2025-12-22T02:54:10Z)
• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• FoundBioNet: A Foundation-Based Model for IDH Genotyping of Glioma from Multi-Parametric MRI [1.4249472316161877]
  We propose a Foundation-based Biomarker Network (FoundBioNet) to noninvasively predict IDH mutation status from multi-parametric MRI.<n>Our model was trained and validated on a diverse, multi-center cohort of 1705 glioma patients from six public datasets.<n>Our model achieved AUCs of 90.58%, 88.08%, 65.41%, and 80.31% on independent test sets from EGD, TCGA, Ivy GAP, RHUH, and UPenn.
  arXiv  Detail & Related papers  (2025-08-09T00:08:10Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• Masked Omics Modeling for Multimodal Representation Learning across Histopathology and Molecular Profiles [0.0]
  Self-supervised learning has driven major advances in computational pathology.<n>However, histopathology alone often falls short for molecular characterization and understanding clinical outcomes.<n>We introduce MORPHEUS, a unified transformer-based pre-training framework that encodes both histopathology and multi-omics data into a shared latent space.
  arXiv  Detail & Related papers  (2025-08-01T15:29:26Z)
• PAST: A multimodal single-cell foundation model for histopathology and spatial transcriptomics in cancer [26.795192024462963]
  PAST is a pan-cancer single-cell foundation model trained on 20 million paired histopathology images and single-cell transcriptomes.<n>It predicts single-cell gene expression, virtual molecular staining, and multimodal survival analysis directly from routine pathology slides.<n>Our work establishes a new paradigm for pathology foundation models, providing a versatile tool for high-resolution spatial omics, mechanistic discovery, and precision cancer research.
  arXiv  Detail & Related papers  (2025-07-08T21:51:25Z)
• Interpretable Graph Kolmogorov-Arnold Networks for Multi-Cancer Classification and Biomarker Identification using Multi-Omics Data [36.92842246372894]
  Multi-Omics Graph Kolmogorov-Arnold Network (MOGKAN) is a deep learning framework that utilizes messenger-RNA, micro-RNA sequences, and DNA methylation samples.<n>By integrating multi-omics data with graph-based deep learning, our proposed approach demonstrates robust predictive performance and interpretability.
  arXiv  Detail & Related papers  (2025-03-29T02:14:05Z)
• UNICORN: A Deep Learning Model for Integrating Multi-Stain Data in Histopathology [2.9389205138207277]
  UNICORN is a multi-modal transformer capable of processing multi-stain histopathology for atherosclerosis severity class prediction. The architecture comprises a two-stage, end-to-end trainable model with specialized modules utilizing transformer self-attention blocks. UNICORN achieved a classification accuracy of 0.67, outperforming other state-of-the-art models.
  arXiv  Detail & Related papers  (2024-09-26T12:13:52Z)
• Multi-Modal and Multi-Attribute Generation of Single Cells with CFGen [76.02070962797794]
  This work introduces CellFlow for Generation (CFGen), a flow-based conditional generative model that preserves the inherent discreteness of single-cell data.<n>CFGen generates whole-genome multi-modal single-cell data reliably, improving the recovery of crucial biological data characteristics.
  arXiv  Detail & Related papers  (2024-07-16T14:05:03Z)
• Co-Heterogeneous and Adaptive Segmentation from Multi-Source and Multi-Phase CT Imaging Data: A Study on Pathological Liver and Lesion Segmentation [48.504790189796836]
  We present a novel segmentation strategy, co-heterogenous and adaptive segmentation (CHASe) We propose a versatile framework that fuses appearance based semi-supervision, mask based adversarial domain adaptation, and pseudo-labeling. CHASe can further improve pathological liver mask Dice-Sorensen coefficients by ranges of $4.2% sim 9.4%$.
  arXiv  Detail & Related papers  (2020-05-27T06:58:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.