Contrastive and Adaptive Multi-modal Masked Autoencoder for Spatial Transcriptomics

• URL: http://arxiv.org/abs/2606.21156v1
• Date: Fri, 19 Jun 2026 06:47:55 GMT
• Title: Contrastive and Adaptive Multi-modal Masked Autoencoder for Spatial Transcriptomics
• Abstract summary: We use partial gene expression to guide the prediction process alongside histology images.<n>We employ a Masked Autoencoder (MAE) to utilize a small fraction of gene expression as genetic anchors for inferring whole-slide gene expression profiles.<n>Our framework consistently surpasses existing methods in both histology-only prediction and spatial imputation.
• Score: 9.067087306213525
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The high cost of spatial transcriptomics (ST) has driven extensive studies into predicting gene expression directly from H&E histology images. However, this prediction task faces an inherent limitation, as tissue morphology alone provides insufficient information to fully resolve underlying gene expression. To address this limitation, a recent study leverages partial gene expression to guide the prediction process alongside histology images. Building on this paradigm, we approach the prediction task as a spatial imputation problem, employing a Masked Autoencoder (MAE) to utilize a small fraction of gene expression as genetic anchors for inferring whole-slide gene expression profiles. Specifically, we propose a bio-saliency score and a learning-to-rank strategy to adaptively identify the most informative spots within the tissue. Based on these identified spots, our framework selects contiguous regions as genetic anchors to ensure suitability for real-world ST profiling hardware. To effectively leverage these anchors, we design a cross-modal joint encoder that integrates visual and genetic modalities. By aligning the selected anchors with their corresponding visual features via contrastive learning, the encoder generates robust joint representations to accurately predict gene expression across the whole slide. Notably, our framework consistently surpasses existing methods in both histology-only prediction and spatial imputation, achieving superior accuracy even without genetic anchors and further excelling with as little as 10% transcriptomic coverage. Our code is available at https://github.com/Kyyle2114/CAMMST.

Related papers

• Hyperbolic Contrastive Learning with Entailment for Spatial Transcriptomics [0.0]
  We propose a hyperbolic contrastive learning model that captures the intrinsic hierarchical relationships within spatial Transcriptomics data.<n>By leveraging hyperbolic geometry and a gene-to-image entailment loss, HyCLoST learns structured, biologically grounded representations that improve gene expression prediction accuracy.
  arXiv  Detail & Related papers  (2026-09-14T18:39:51Z)
• BioM-JEPA: joint-embedding prediction of graph-connected gene blocks in single cells [64.21963170814706]
  BioM-JEPA predicts aggregate representations of graph-connected gene blocks.<n>A student network infers each target-block representation from the remaining genes in a cell.<n>Across CellBench tasks, BioM-JEPA embeddings retained expression, pathway and neighbourhood information.
  arXiv  Detail & Related papers  (2026-08-06T11:58:29Z)
• Dual-Path Knowledge-Augmented Contrastive Alignment Network for Spatially Resolved Transcriptomics [5.455957568203595]
  High cost has driven efforts to predict spatial gene expression from whole slide images.<n>Current methods face significant limitations, such as under-exploitation of high-level biological context.<n>We propose DKAN, a novel Dual-path Knowledge-Augmented contrastive alignment Network.
  arXiv  Detail & Related papers  (2025-11-21T10:58:04Z)
• Learning from Gene Names, Expression Values and Images: Contrastive Masked Text-Image Pretraining for Spatial Transcriptomics Representation Learning [2.234171233882506]
  CoMTIP is a Contrastive Masked Text-Image Pretraining framework.<n>It learns from images, gene names, and expression values while capturing fine-grained visual context for spatial transcriptomics.<n>CoMTIP not only surpasses previous methods on diverse downstream tasks but also achieves zero-shot gene expression prediction.
  arXiv  Detail & Related papers  (2025-09-21T03:05:17Z)
• A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics [8.854289521774483]
  HESCAPE is a benchmark for evaluating cross-modal contrastive pretraining in spatial transcriptomics.<n>Gene models pretrained on spatial transcriptomics data outperform both those trained without spatial data and simple baseline approaches.<n>We identify batch effects as a key factor that interferes with effective cross-modal alignment.
  arXiv  Detail & Related papers  (2025-08-02T21:11:36Z)
• GRAPE: Heterogeneous Graph Representation Learning for Genetic Perturbation with Coding and Non-Coding Biotype [51.58774936662233]
  Building gene regulatory networks (GRN) is essential to understand and predict the effects of genetic perturbations.<n>In this work, we leverage pre-trained large language model and DNA sequence model to extract features from gene descriptions and DNA sequence data.<n>We introduce gene biotype information for the first time in genetic perturbation, simulating the distinct roles of genes with different biotypes in regulating cellular processes.
  arXiv  Detail & Related papers  (2025-05-06T03:35:24Z)
• Multimodal contrastive learning for spatial gene expression prediction using histology images [13.47034080678041]
  We propose textbfmclSTExp, a multimodal contrastive learning with Transformer and Densenet-121 encoder for Spatial Transcriptomics Expression prediction. textbfmclSTExp has superior performance in predicting spatial gene expression. It has shown promise in interpreting cancer-specific overexpressed genes, elucidating immune-related genes, and identifying specialized spatial domains annotated by pathologists.
  arXiv  Detail & Related papers  (2024-07-11T06:33:38Z)
• Predicting Genetic Mutation from Whole Slide Images via Biomedical-Linguistic Knowledge Enhanced Multi-label Classification [119.13058298388101]
  We develop a Biological-knowledge enhanced PathGenomic multi-label Transformer to improve genetic mutation prediction performances. BPGT first establishes a novel gene encoder that constructs gene priors by two carefully designed modules. BPGT then designs a label decoder that finally performs genetic mutation prediction by two tailored modules.
  arXiv  Detail & Related papers  (2024-06-05T06:42:27Z)
• VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling [60.91599380893732]
  VQDNA is a general-purpose framework that renovates genome tokenization from the perspective of genome vocabulary learning. By leveraging vector-quantized codebooks as learnable vocabulary, VQDNA can adaptively tokenize genomes into pattern-aware embeddings.
  arXiv  Detail & Related papers  (2024-05-13T20:15:03Z)
• Efficient and Scalable Fine-Tune of Language Models for Genome Understanding [49.606093223945734]
  We present textscLingo: textscLanguage prefix ftextscIne-tuning for textscGentextscOmes. Unlike DNA foundation models, textscLingo strategically leverages natural language foundation models' contextual cues. textscLingo further accommodates numerous downstream fine-tune tasks by an adaptive rank sampling method.
  arXiv  Detail & Related papers  (2024-02-12T21:40:45Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.