Concept-based explanation of gene expression prediction from H&E images

• URL: http://arxiv.org/abs/2608.16669v1
• Date: Mon, 17 Aug 2026 14:58:56 GMT
• Title: Concept-based explanation of gene expression prediction from H&E images
• Abstract summary: We develop a ViT-based framework for virtual transcriptomics from H&E images.<n>Our architecture accurately predicts clinically relevant ST signatures and accompanying molecular phenotypes.<n>Our framework is readily applicable to a broad range of ViT-based pathology models.
• Score: 20.27953548393374
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advances in pathology foundation models have enabled accurate prediction of spatial transcriptomics (ST) from routine H&E images. However, existing explainability methods for vision transformer (ViT)-based models are largely limited to local heatmaps and do not reveal how morphological concepts contribute to ST predictions. Here, we introduce an explainable framework that combines relevance propagation and concept discovery to link transcriptional programs to tissue morphology. We developed a ViT-based framework for virtual ST from H&E images that combines ViT-aware layer-wise relevance propagation with relaxed archetypal TopK sparse autoencoder-based concept discovery. This approach provides both local explanations and global insights into the morphological patterns associated with transcriptional programs. We applied the framework to colorectal cancer ST data from the HEST-1k cohort and evaluated its generalizability in TCGA COAD. Our architecture accurately predicts clinically relevant ST signatures and accompanying molecular phenotypes. Measured and predicted gene expression profiles reveal substantial spatial heterogeneity of the colorectal cancer subtypes iCMS2 and iCMS3 across a large number of samples. Spatially resolved and aggregated iCMS classification achieve weighted F1 scores of 0.872 and 0.819 (0.770 in TCGA COAD), respectively, and both stratify patient outcome. Beyond prediction, our framework establishes a relevance-based concept atlas linking molecular phenotypes to histopathological representations. Comparison of activation- with relevance-derived concepts demonstrates that relevances provide a more direct link between tissue morphology and downstream predictions. We establish a general strategy for concept-based explanation of spatial prediction, and our framework is readily applicable to a broad range of ViT-based pathology models.

Related papers

• STP-BENCH: A Unified Systematic Benchmark for Virtual Spatial Transcriptomics from Histopathology Images [8.906169973742758]
  Computational approaches that predict gene expression directly from hematoxylin and eosin slides, termed virtual ST, have rapidly emerged.<n>We present-BENCH, a standardized benchmark for virtual ST models.<n>We evaluate 21 approaches, re-implemented with a unified pathology model as the morphological encoder when architecturally applicable.
  arXiv  Detail & Related papers  (2026-09-05T07:43:36Z)
• Understanding Synergistic Interactions among Pathology Foundation Models via Adaptive Fusion [49.550545038402184]
  Pathology foundation models (PFMs) provide strong tile-level representations via self-supervised pre-training on large-scale pathology images.<n>We present AdaFusion, a lightweight adaptive fusion framework that integrates complementary signals from multiple frozen PFMs.<n>AdaFusion consistently outperforms individual PFMs and other fusion baselines, while providing interpretable tissue visualisation.
  arXiv  Detail & Related papers  (2026-08-02T16:44:45Z)
• PathAR: Structure-First Autoregressive Synthesis of Multimodal Pathology Images [51.428093790826814]
  We propose pathology Autorgressive modeling (PathAR) to factorize structure and appearance for modality-label-conditioned pathology generation.<n>PathAR stabilizes morphology under heterogeneous modality-specific appearances and enables spatially aligned image-mask pair generation.
  arXiv  Detail & Related papers  (2026-06-01T01:43:48Z)
• Multimodal Graph-based Classification of Esophageal Motility Disorders [73.90451172929117]
  Diagnosing esophageal motility disorders pose significant challenges due to the complexity of high-resolution impedance manometry data and variability in clinical interpretation.<n>This work explores the feasibility of a multimodal Machine Learning-based classification approach that combines HRIM recordings with patient-specific information and incorporates a graph-based modeling of esophageal physiology.
  arXiv  Detail & Related papers  (2026-05-13T14:52:12Z)
• Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation [6.938242893061667]
  We present SCOUT: Semantic Context-aware mOdality fUsion Transformer, a context-aware concept-grounded multimodal framework for pathology report generation.<n>Method integrates local histological patterns, whole-slide context, and expert-curated semantic descriptors within a unified learning paradigm.<n>By combining depth-aware contextual modulation with adaptive multimodal fusion during text generation, the framework produces clinically coherent reports.
  arXiv  Detail & Related papers  (2026-05-01T22:40:24Z)
• Cross-Slice Knowledge Transfer via Masked Multi-Modal Heterogeneous Graph Contrastive Learning for Spatial Gene Expression Inference [7.79162872496708]
  SpaHGC is a graph-based model that captures both intra-slice and inter-slice spot-spot relationships from histology images.<n>It integrates local spatial context within the target slide and cross-slide similarities computed from image embeddings extracted by a pathology foundation model.<n>SpaHGC significantly outperforms the existing nine state-of-the-art methods across all evaluation metrics.
  arXiv  Detail & Related papers  (2026-03-24T05:49:00Z)
• Class Visualizations and Activation Atlases for Enhancing Interpretability in Deep Learning-Based Computational Pathology [1.5921492981295717]
  Class visualizations (CVs) and activation atlases (AAs) were evaluated for a transformer-based foundation model.<n>CVs preserved recognizability for morphologically distinct tissues but showed reduced separability for overlapping cancer subclasses.<n>AAs revealed layer-dependent organization: coarse tissue-level concepts formed coherent regions, whereas finer subclasses exhibited dispersion and overlap.
  arXiv  Detail & Related papers  (2026-03-07T12:23:10Z)
• Towards Spatial Transcriptomics-driven Pathology Foundation Models [32.70436266943553]
  We introduce a vision-omics self-supervised learning framework that infuses localized molecular information into pathology vision encoders.<n>We instantiate SEAL by training on over 700,000 paired gene expression spot-tissue region examples spanning tumor and normal samples from 14 organs.<n> SEAL encoders exhibit robust domain generalization on out-of-distribution and enable new cross-modal capabilities such as gene-to-image retrieval.
  arXiv  Detail & Related papers  (2026-02-15T15:06:45Z)
• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• AdaFusion: Prompt-Guided Inference with Adaptive Fusion of Pathology Foundation Models [49.550545038402184]
  We propose AdaFusion, a novel prompt-guided inference framework.<n>Our method compresses and aligns tile-level features from diverse models.<n>AdaFusion consistently surpasses individual PFMs across both classification and regression tasks.
  arXiv  Detail & Related papers  (2025-08-07T07:09:31Z)
• Unsupervised ensemble-based phenotyping helps enhance the discoverability of genes related to heart morphology [57.25098075813054]
  We propose a new framework for gene discovery entitled Un Phenotype Ensembles. It builds a redundant yet highly expressive representation by pooling a set of phenotypes learned in an unsupervised manner. These phenotypes are then analyzed via (GWAS), retaining only highly confident and stable associations.
  arXiv  Detail & Related papers  (2023-01-07T18:36:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.