A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction

• URL: http://arxiv.org/abs/2604.03630v1
• Date: Sat, 04 Apr 2026 08:00:26 GMT
• Title: A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction
• Abstract summary: We present textbfours (textbfSpatial textbfTranscriptomics and histtextbfOlogy textbfRepresentation textbfModel), a foundation model trained on 1.2 million spatially resolved transcriptomic profiles with matched histology across 18 organs.
• Score: 13.466764029232898
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spatial transcriptomics (ST) enables gene expression mapping within anatomical context but remains costly and low-throughput. Hematoxylin and eosin (H\&E) staining offers rich morphology yet lacks molecular resolution. We present \textbf{\ours} (\textbf{S}patial \textbf{T}ranscriptomics and hist\textbf{O}logy \textbf{R}epresentation \textbf{M}odel), a foundation model trained on 1.2 million spatially resolved transcriptomic profiles with matched histology across 18 organs. Using a hierarchical architecture integrating morphological features, gene expression, and spatial context, STORM bridges imaging and omics through robust molecular--morphological representations. STORM enhances spatial domain discovery, producing biologically coherent tissue maps, and outperforms existing methods in predicting spatial gene expression from H\&E images across 11 tumor types. The model is platform-agnostic, performing consistently across Visium, Xenium, Visium HD, and CosMx. Applied to 23 independent cohorts comprising 7,245 patients, STORM significantly improves immunotherapy response prediction and prognostication over established biomarkers, providing a scalable framework for spatially informed discovery and clinical precision medicine.

Related papers

• Learning latent progression states from spatial heterogeneity in uterine histopathology [27.352389544902163]
  SpaTIE is a uterus-specific computational pathology framework that learns morphology-aware representations.<n>SpaTIE inferred tumor-state axes from cross-sectional morphology without temporal or molecular supervision.<n> Integrative multi-omics analyses linked the inferred states to DNA methylation, somatic copy-number variation, mutation, RNA-seq and RPPA profiles.
  arXiv  Detail & Related papers  (2026-08-18T03:56:17Z)
• JASPR: Joint Spatial Representation learning of histology and spatial genomics for improved virtual genomic screening and clinical prognostication [1.8440626395175606]
  We present JASPR (Joint Spatial Representation learning), a self-supervised deep learning framework that integrates HE images and ST data through a cross-modal reconstruction objective.<n>It employs shared modules to capture universal spatial properties across modalities, while modality-specific experts encode features unique to morphological and genomic data.<n>We train and validate JASPR on breast cancer datasets, demonstrating that its learned joint representation substantially improves HE-based prediction of 9,248 genes and provides prognostic value for breast cancer outcomes.
  arXiv  Detail & Related papers  (2026-06-24T00:35:49Z)
• Spatial Transcriptomics-Guided Alignment Enhances Molecular Profiling in Pathology Foundation Model [43.24445461533517]
  We present a spatial transcriptomics-guided alignment framework for Molecular Profiling (STAMP)<n>STAMP applies a pathway-informed alignment strategy that aggregates transcriptomic data into biologically functional pathways.<n>The clinical utility of these augmented representations was validated through a multi-tier evaluation framework.
  arXiv  Detail & Related papers  (2026-05-29T16:41:14Z)
• Towards Spatial Transcriptomics-driven Pathology Foundation Models [32.70436266943553]
  We introduce a vision-omics self-supervised learning framework that infuses localized molecular information into pathology vision encoders.<n>We instantiate SEAL by training on over 700,000 paired gene expression spot-tissue region examples spanning tumor and normal samples from 14 organs.<n> SEAL encoders exhibit robust domain generalization on out-of-distribution and enable new cross-modal capabilities such as gene-to-image retrieval.
  arXiv  Detail & Related papers  (2026-02-15T15:06:45Z)
• Fusing Pixels and Genes: Spatially-Aware Learning in Computational Pathology [46.83014413674925]
  STAMP is a spatial transcriptomics-augmented multimodal pathology representation learning framework.<n>Our study shows that self-supervised, gene-guided training provides a robust and task-agnostic signal for learning pathology image representations.<n>We validate STAMP across six datasets and four downstream tasks, where it consistently achieves strong performance.
  arXiv  Detail & Related papers  (2026-02-15T00:59:13Z)
• R-GenIMA: Integrating Neuroimaging and Genetics with Interpretable Multimodal AI for Alzheimer's Disease Progression [63.97617759805451]
  Early detection of Alzheimer's disease requires models capable of integrating macro-scale neuroanatomical alterations with micro-scale genetic susceptibility.<n>We introduce R-GenIMA, an interpretable multimodal large language model that couples a novel ROI-wise vision transformer with genetic prompting.<n>R-GenIMA achieves state-of-the-art performance in four-way classification across normal cognition, subjective memory concerns, mild cognitive impairment, and AD.
  arXiv  Detail & Related papers  (2025-12-22T02:54:10Z)
• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• HiFusion: Hierarchical Intra-Spot Alignment and Regional Context Fusion for Spatial Gene Expression Prediction from Histopathology [7.982889842329205]
  HiFusion is a novel deep learning framework that integrates two complementary components.<n>We show that HiFusion achieves state-of-the-art performance across both 2D slide-wise cross-validation and more challenging 3D sample-specific scenarios.<n>These results underscore HiFusion's potential as a robust, accurate, and scalable solution for ST inference from routine histopathology.
  arXiv  Detail & Related papers  (2025-11-17T04:47:39Z)
• PH2ST:ST-Prompt Guided Histological Hypergraph Learning for Spatial Gene Expression Prediction [9.420121324844066]
  We propose PH2ST, a prompt-guided hypergraph learning framework, to guide multi-scale histological representation learning for spatial gene expression prediction.<n> PH2ST not only outperforms existing state-of-the-art methods, but also shows strong potential for practical applications such as imputing missing spots, ST super-resolution, and local-to-global prediction.
  arXiv  Detail & Related papers  (2025-03-21T03:10:43Z)
• MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention [57.044719143401664]
  Histopathology and transcriptomics are fundamental modalities in oncology, encapsulating the morphological and molecular aspects of the disease.<n>We present MIRROR, a novel multi-modal representation learning method designed to foster both modality alignment and retention.<n>Extensive evaluations on TCGA cohorts for cancer subtyping and survival analysis highlight MIRROR's superior performance.
  arXiv  Detail & Related papers  (2025-03-01T07:02:30Z)
• TopoTxR: A topology-guided deep convolutional network for breast parenchyma learning on DCE-MRIs [49.69047720285225]
  We propose a novel topological approach that explicitly extracts multi-scale topological structures to better approximate breast parenchymal structures. We empirically validate emphTopoTxR using the VICTRE phantom breast dataset. Our qualitative and quantitative analyses suggest differential topological behavior of breast tissue in treatment-na"ive imaging.
  arXiv  Detail & Related papers  (2024-11-05T19:35:10Z)
• Multi-modal Spatial Clustering for Spatial Transcriptomics Utilizing High-resolution Histology Images [1.3124513975412255]
  spatial transcriptomics (ST) enables transcriptome-wide gene expression profiling while preserving spatial context. Current spatial clustering methods fail to fully integrate high-resolution histology image features with gene expression data. We propose a novel contrastive learning-based deep learning approach that integrates gene expression data with histology image features.
  arXiv  Detail & Related papers  (2024-10-31T00:32:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.