Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction

• URL: http://arxiv.org/abs/2606.29949v1
• Date: Mon, 29 Jun 2026 08:24:53 GMT
• Title: Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction
• Abstract summary: We show that training a lightweight alignment module atop frozen histopathology and RNA-Seq foundation models enables open-vocabulary molecular prompting.<n>We achieve a 25-fold improvement in retrieval over baseline methods.<n>We validate clinical utility on the POSEIDON clinical trial.
• Score: 3.229566379058554
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: H&E-stained whole-slide images offer cohort-scale availability and rich spatial context but lack molecular specificity, whereas bulk RNA-seq provides transcriptome-wide resolution at high cost with limited archival availability. We show that training a lightweight alignment module atop frozen histopathology and RNA-Seq foundation models enables open-vocabulary molecular prompting -- querying H&E slides with gene-set signatures to predict pathway activity without sequencing or end-to-end retraining. Using contrastive learning on a multi-cancer cohort (N=1,720), we achieve a 25-fold improvement in retrieval over baseline methods. Systematic analysis reveals a graduated predictability spectrum: morphologically grounded programs (cell-cycle programs, immune-related) are most reliably predicted (R^2>0.5), while predicting pathways with no morphological footprint remains challenging as expected. We validate clinical utility on the POSEIDON clinical trial: H&E-predicted squamous cell carcinoma scores recapitulate NSCLC subtype identity and predicted IFN-gamma mirror PD-L1 tumor-cell expression groups. Furthermore, genesets describing immune activation and fibrosis predict known tumor microenvironment archetypes from histology alone. We further validate generalization of our approach across unseen cohorts and demonstrate data-efficient domain adaptation, establishing a slide-native framework for molecular analysis on H&E images.

Related papers

• Benchmarking Machine Learning Models for Multi-Omics-Based Breast Cancer Prediction [0.0]
  Estrogen Receptor (ER) status is a critical biomarker in breast cancer diagnosis, prognosis, and treatment selection.<n>This study presents a systematic benchmarking analysis of classical machine learning models for ER status prediction.
  arXiv  Detail & Related papers  (2026-06-27T02:46:40Z)
• Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology [47.79180201495836]
  MixTIME is a multimodal foundation model that integrates pathology foundation models trained across distinct modalities.<n>It predicts multiplex immunofluorescence (mIF) protein expression from hematoxylin and eosin whole-slide images.<n>MixTIME achieves state-of-the-art performance across 17 protein markers as measured by correlation metrics.
  arXiv  Detail & Related papers  (2026-06-16T16:22:42Z)
• A physics-informed foundation model for quantitative diffusion MRI [46.74037857287099]
  A physics-informed generative microstructure network (PIGMENT) learns a universal generative prior to human brain microstructure and adapts it zero-shot to each participant's measured data to recover subject-specific maps.<n>PIGMENT estimates demonstrated strong biological validity, preserving submillimeter cortical microarchitectural patterns and early-childhood white matter developmental trajectories from 10-fold accelerated scans.<n>These results establish PIGMENT as a physics-informed foundation model that extends quantitative diffusion MRI into regimes traditionally too sparse, heterogeneous, or clinically constrained for reliable analysis.
  arXiv  Detail & Related papers  (2026-05-29T07:26:35Z)
• Transcriptomic Models for Immunotherapy Response Prediction Show Limited Cross-cohort Generalisability [15.181294821978902]
  We benchmark nine state-of-the-art transcriptomic ICI response predictors, five bulk RNA-seq-based models, and four scRNA-seq-based models.<n>Overall, predictive performance was modest; bulk RNA-seq models performed at or near chance level across most cohorts.<n> Pathway-level analyses revealed sparse and inconsistent biomarker signals across models.
  arXiv  Detail & Related papers  (2026-04-07T06:18:59Z)
• Investigating the Impact of Histopathological Foundation Models on Regressive Prediction of Homologous Recombination Deficiency [52.50039435394964]
  We systematically evaluate foundation models for regression-based tasks.<n>We extract patch-level features from whole slide images (WSI) using five state-of-the-art foundation models.<n>Models are trained to predict continuous HRD scores based on these extracted features across breast, endometrial, and lung cancer cohorts.
  arXiv  Detail & Related papers  (2026-01-29T14:06:50Z)
• Genome-Anchored Foundation Model Embeddings Improve Molecular Prediction from Histology Images [22.274588547000352]
  PathLUPI uses transcriptomic privileged information during training to extract genome-anchored histological embeddings.<n>It achieves AUC $geq$ 0.80 in 14 of the biomarker prediction and molecular subtyping tasks and C-index $geq$ 0.70 in survival cohorts of 5 major cancer types.
  arXiv  Detail & Related papers  (2025-06-24T14:48:12Z)
• ImmunoDiff: A Diffusion Model for Immunotherapy Response Prediction in Lung Cancer [10.797150801746957]
  Accurately predicting immunotherapy response in Non-Small Cell Lung Cancer (NSCLC) remains a critical unmet need.<n>Existing radiomics and deep learning-based predictive models rely primarily on pre-treatment imaging to predict categorical response outcomes.<n>This study introduces ImmunoDiff, an anatomy-aware diffusion model designed to synthesize post-treatment CT scans from baseline imaging while incorporating clinically relevant constraints.
  arXiv  Detail & Related papers  (2025-05-29T17:19:40Z)
• Interpretable Graph Kolmogorov-Arnold Networks for Multi-Cancer Classification and Biomarker Identification using Multi-Omics Data [36.92842246372894]
  Multi-Omics Graph Kolmogorov-Arnold Network (MOGKAN) is a deep learning framework that utilizes messenger-RNA, micro-RNA sequences, and DNA methylation samples.<n>By integrating multi-omics data with graph-based deep learning, our proposed approach demonstrates robust predictive performance and interpretability.
  arXiv  Detail & Related papers  (2025-03-29T02:14:05Z)
• TopoTxR: A topology-guided deep convolutional network for breast parenchyma learning on DCE-MRIs [49.69047720285225]
  We propose a novel topological approach that explicitly extracts multi-scale topological structures to better approximate breast parenchymal structures. We empirically validate emphTopoTxR using the VICTRE phantom breast dataset. Our qualitative and quantitative analyses suggest differential topological behavior of breast tissue in treatment-na"ive imaging.
  arXiv  Detail & Related papers  (2024-11-05T19:35:10Z)
• Lung Cancer Lesion Detection in Histopathology Images Using Graph-Based Sparse PCA Network [93.22587316229954]
  We propose a graph-based sparse principal component analysis (GS-PCA) network, for automated detection of cancerous lesions on histological lung slides stained by hematoxylin and eosin (H&E) We evaluate the performance of the proposed algorithm on H&E slides obtained from an SVM K-rasG12D lung cancer mouse model using precision/recall rates, F-score, Tanimoto coefficient, and area under the curve (AUC) of the receiver operator characteristic (ROC)
  arXiv  Detail & Related papers  (2021-10-27T19:28:36Z)
• Transcriptome-wide prediction of prostate cancer gene expression from histopathology images using co-expression based convolutional neural networks [0.8874479658912061]
  We propose a new, computationally efficient approach for disease specific modelling of relationships between morphology and gene expression. We conducted the first transcriptome-wide analysis in prostate cancer, using CNNs to predict bulk RNA-sequencing estimates.
  arXiv  Detail & Related papers  (2021-04-19T13:50:25Z)
• A Systematic Approach to Featurization for Cancer Drug Sensitivity Predictions with Deep Learning [49.86828302591469]
  We train >35,000 neural network models, sweeping over common featurization techniques. We found the RNA-seq to be highly redundant and informative even with subsets larger than 128 features.
  arXiv  Detail & Related papers  (2020-04-30T20:42:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.