Linking spatial biology and clinical histology via Haiku

• URL: http://arxiv.org/abs/2605.00925v1
• Date: Thu, 30 Apr 2026 18:53:32 GMT
• Title: Linking spatial biology and clinical histology via Haiku
• Abstract summary: We present Haiku, a tri-modal contrastive learning model trained on multiplexed immunofluorescence (mIF)<n>It comprises 26.7 million spatial patches from 3,218 tissue sections across 1,606 patients spanning 11 organ types.
• Score: 35.723756132708026
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Integrating molecular, morphological, and clinical data is essential for basic and translational biomedical research, yet systematic frameworks for jointly modeling these modalities remain limited. Here we present Haiku, a tri-modal contrastive learning model trained on multiplexed immunofluorescence (mIF). It comprises 26.7 million spatial proteomics patches from 3,218 tissue sections across 1,606 patients spanning 11 organ types, with matched hematoxylin and eosin (H&E) histology and clinical metadata aligned in a shared embedding space. Haiku enables three-way cross-modal retrieval, improves downstream classification and clinical prediction tasks over unimodal baselines, and supports zero-shot biomarker inference through fusion retrieval conditioned on clinical metadata-only text descriptions. Across tasks, Haiku outperforms competing approaches, achieving cross-modal retrieval (Recall@50 up to 0.611 versus near-zero baseline), survival prediction (C-index 0.737, +7.91% relative improvement), and zero-shot biomarker inference (mean Pearson correlation 0.718 across 52 biomarkers). Furthermore, we introduce a counterfactual prediction framework in which modifying only clinical metadata while fixing tissue morphology surfaces niche-specific molecular shifts associated with breast cancer stage progression and lung cancer survival outcomes. In a lung adenocarcinoma case study, the counterfactual analysis recovers niche-specific shifts characterized by increased CD8 and granzyme B, reduced PD-L1, and decreased Ki67, broadly consistent with patterns reported for favorable outcomes. We present these counterfactual results as exploratory, hypothesis-generating signals rather than mechanistic claims. These capabilities demonstrate that tri-modal alignment via Haiku enables integrative analysis of spatial biology, bridging molecular measurements with clinical context for biological exploration.

Related papers

• A Multimodal Foundation Model for Longitudinal Patient Representation and Scalable Insight Generation in Oncology [2.4534049320113227]
  We introduce the oFM, a foundation model developed on a real-world oncology cohort of 1.67 million cancer patients.<n>The oFM encodes daily clinical and molecular episodes and integrates them over time to produce a patient state embedding.<n>In prognostic benchmarks, the oFM improved AUC for treatment response, progression-free survival, and overall survival.
  arXiv  Detail & Related papers  (2026-08-25T15:17:12Z)
• Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction [3.229566379058554]
  We show that training a lightweight alignment module atop frozen histopathology and RNA-Seq foundation models enables open-vocabulary molecular prompting.<n>We achieve a 25-fold improvement in retrieval over baseline methods.<n>We validate clinical utility on the POSEIDON clinical trial.
  arXiv  Detail & Related papers  (2026-06-29T08:24:53Z)
• Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology [47.79180201495836]
  MixTIME is a multimodal foundation model that integrates pathology foundation models trained across distinct modalities.<n>It predicts multiplex immunofluorescence (mIF) protein expression from hematoxylin and eosin whole-slide images.<n>MixTIME achieves state-of-the-art performance across 17 protein markers as measured by correlation metrics.
  arXiv  Detail & Related papers  (2026-06-16T16:22:42Z)
• MARVEL: Universal Murray's Law-informed Vessel Tree Segmentation and Topology Estimation [75.40785674602147]
  Deep learning methods for vascular segmentation often neglect biophysical constraints.<n>This leads to physiologically implausible branching and misclassification vascular trees.<n>We introduce MARVEL, a backbone-agnostic framework that integrates biophysical priors into vascular tree extraction.
  arXiv  Detail & Related papers  (2026-05-25T02:32:29Z)
• Simulating clinical interventions with a generative multimodal model of human physiology [49.828582089394445]
  HealthFormer is a decoder-only transformer that models the human physiological trajectory generatively.<n>We tokenise each participant's health trajectory across 667 measurements spanning seven domains.<n>HealthFormer improves prediction for 27 of 30 incident-disease and mortality endpoints, exceeding established clinical risk scores in every comparison.
  arXiv  Detail & Related papers  (2026-04-30T14:10:30Z)
• Multimodal Fusion of Histopathology Images and Electronic Health Records for Early Breast Cancer Diagnosis [15.70912899013153]
  This paper presents a systematic framework that integrates patch-level histopathology features with structured clinical data.<n>We train and evaluate unimodal image models (a simple CNN baseline and ResNet-18 with transfer learning) and intermediate-fusion models (XGBoost and a multilayer perceptron)<n>Our results demonstrate that multimodal integration delivers meaningful improvements in both predictive performance and clinical transparency.
  arXiv  Detail & Related papers  (2026-04-18T19:46:34Z)
• A Generative Foundation Model for Multimodal Histopathology [25.341102846996453]
  We introduce MuPD, a generative foundation model that embeds hematoxylin and eosin (H&E)-stained histology, molecular RNA profiles, and clinical text into a shared latent space.<n>MuPD supports diverse cross-modal tasks with minimal or no task-specific fine-tuning.<n>For text-conditioned and image-to-image generation, MuPD synthesizes histologically faithful tissue architectures.
  arXiv  Detail & Related papers  (2026-04-04T08:09:11Z)
• Investigating the Impact of Histopathological Foundation Models on Regressive Prediction of Homologous Recombination Deficiency [52.50039435394964]
  We systematically evaluate foundation models for regression-based tasks.<n>We extract patch-level features from whole slide images (WSI) using five state-of-the-art foundation models.<n>Models are trained to predict continuous HRD scores based on these extracted features across breast, endometrial, and lung cancer cohorts.
  arXiv  Detail & Related papers  (2026-01-29T14:06:50Z)
• ModalSurv: A Multimodal Deep Survival Framework for Prostate and Bladder Cancer [5.509924404430891]
  We present ModaliSurv, a multimodal deep survival model utilising DeepHit with a projection layer and inter-modality cross-attention.<n>The model is designed to capture complementary prognostic signals across modalities and estimate individualised time-to-biochemical recurrence.
  arXiv  Detail & Related papers  (2025-09-05T11:52:53Z)
• Predicting Neoadjuvant Chemotherapy Response in Triple-Negative Breast Cancer Using Pre-Treatment Histopathologic Images [2.23127246021293]
  Triple-negative breast cancer (TNBC) remains a major clinical challenge due to its aggressive behavior and lack of targeted therapies.<n>We present an attention-based multiple instance learning framework designed to predict pathologic complete response (pCR) directly from pre-treatment hematoxylin and eosin (H&E)-stained biopsy slides.
  arXiv  Detail & Related papers  (2025-05-20T02:06:34Z)
• TopoTxR: A topology-guided deep convolutional network for breast parenchyma learning on DCE-MRIs [49.69047720285225]
  We propose a novel topological approach that explicitly extracts multi-scale topological structures to better approximate breast parenchymal structures. We empirically validate emphTopoTxR using the VICTRE phantom breast dataset. Our qualitative and quantitative analyses suggest differential topological behavior of breast tissue in treatment-na"ive imaging.
  arXiv  Detail & Related papers  (2024-11-05T19:35:10Z)
• HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Models [11.03503711055749]
  HONeYBEE is an open-source framework that integrates multimodal biomedical data for oncology applications.<n>It processes clinical data (structured and unstructured), whole-slide images, radiology scans, and molecular profiles to generate unified patient-level embeddings.<n>These embeddings enable survival prediction, cancer-type classification, patient similarity retrieval, and cohort clustering.
  arXiv  Detail & Related papers  (2024-05-13T04:35:14Z)
• Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction [43.1748594898772]
  We propose a multimodal transformer (PathOmics) integrating pathology and genomics insights into colon-related cancer survival prediction. We emphasize the unsupervised pretraining to capture the intrinsic interaction between tissue microenvironments in gigapixel whole slide images. We evaluate our approach on both TCGA colon and rectum cancer cohorts, showing that the proposed approach is competitive and outperforms state-of-the-art studies.
  arXiv  Detail & Related papers  (2023-07-22T00:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.