Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining

• URL: http://arxiv.org/abs/2605.16392v1
• Date: Tue, 12 May 2026 06:49:35 GMT
• Title: Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining
• Abstract summary: We introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the projection layer.<n>MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features.<n>MIST improves 240 of 256 configurations over the standard projection layer, with an average gain of +3.5%, observed consistently across endpoint types.
• Score: 21.47707112793145
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator. While encoders and aggregators have been extensively studied, the projection layer remains a largely morphology-only bottleneck. This limits endpoints such as biomarker status and survival, which are governed by a molecular state that is not fully captured by H&E morphology. We introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains. MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes. It requires no transcriptomics at inference and can be inserted before standard MIL aggregators. We evaluate MIST across 23 downstream tasks and 8 MIL aggregators. MIST improves 240 of 256 configurations over the standard projection layer, with an average gain of +3.5%, observed consistently across endpoint types: +5.2% on survival prediction, +3.3% on tissue subtyping, and +2.6% on biomarker prediction. Ablations confirm that gene-derived prototypes are the primary source of the gains, while spatial, biological, and pathological analyses show that cross-modal prototype affinities capture spatially coherent molecular programs from H&E alone.

Related papers

• Fixed-Dimensional Latent Flow for Generating Variable-Size 3D Molecules [28.11511793703245]
  We introduce a two-stage generative framework that relies entirely on a single fixed-dimensional molecule-level latent representation to generate variable-size molecules.<n>On PCQM4Mv2, EF-TALFM achieves the highest fraction of molecules that are unique, training-set novel, pass sanitization and PoseBusters sanity checks, 89.4%, compared with 75.6% for UAE-3D and 69.8% for FlowMol.
  arXiv  Detail & Related papers  (2026-09-08T07:03:51Z)
• Canopy: A Heterograph Foundation Model for Metabolic Engineering [0.0]
  We present Canopy, a heterogeneous graph foundation model that integrates ten public and proprietary data sources into a unified knowledge graph.<n> Node features are encoded through domain-specific foundation models, yielding a multi-modal representation within a single graph.<n>On the downstream task of fermentation titer prediction, frozen Canopy embeddings achieve $R2 = 0.41$ with a lightweight probe.
  arXiv  Detail & Related papers  (2026-07-07T12:48:38Z)
• Biological Spatial Priors Regularize Foundation Model Representations for Cross-Site MSI Generalization in Colorectal Cancer [0.0]
  We introduce a biologically motivated spatial prior based on peripheral distance encoding.<n>We evaluate six foundation model and MIL aggregator combinations as a reference, then assess the effect of each spatial prior.<n>Results suggest biologically grounded spatial priors act as regularizers that reduce reliance on site-specific imaging patterns.
  arXiv  Detail & Related papers  (2026-05-04T14:39:40Z)
• Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathology [14.800356715603087]
  Multiple instance learning (MIL) has enabled substantial progress in computational histopathology.<n>Heatmaps are widely used to validate MIL models and to discover tissue biomarkers.<n>We introduce a general framework for evaluating the quality of MIL heatmaps without requiring additional labels.
  arXiv  Detail & Related papers  (2026-03-09T12:45:48Z)
• MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Models [5.15189458127282]
  We propose MINT, a fine-tuning framework that incorporates spatial transcriptomics supervision into pretrained pathology Vision Transformers.<n> MINT appends a learnable ST token to the ViT input to encode transcriptomic information separately from the morphological CLS token.<n> MINT achieves the best overall performance on both HEST-Bench for gene expression prediction and general pathology tasks.
  arXiv  Detail & Related papers  (2026-03-09T02:29:10Z)
• Breaking the Modality Barrier: Generative Modeling for Accurate Molecule Retrieval from Mass Spectra [60.08608779794957]
  We propose GLMR, a Generative Language Model-based Retrieval framework.<n>In the pre-retrieval stage, a contrastive learning-based model identifies top candidate molecules as contextual priors for the input mass spectrum.<n>In the generative retrieval stage, these candidate molecules are integrated with the input mass spectrum to guide a generative model in producing refined molecular structures.
  arXiv  Detail & Related papers  (2025-11-09T07:25:53Z)
• MMAP: A Multi-Magnification and Prototype-Aware Architecture for Predicting Spatial Gene Expression [1.083137038945176]
  Spatial Transcriptomics (ST) enables the measurement of gene expression while preserving spatial information.<n>Recent developments have explored the use of hematoxylin and eosin (H&E)-stained whole-slide images (WSIs) to predict transcriptome-wide gene expression profiles through deep neural networks.<n>However, predicting spatial gene expression from histological images remains a challenging problem due to the significant modality gap between visual features and molecular signals.<n>In this work, we propose a novel framework, MMAP (Multi-MAgnification and Prototype-enhanced architecture), that addresses both challenges simultaneously.
  arXiv  Detail & Related papers  (2025-10-13T12:41:09Z)
• Aligned Manifold Property and Topology Point Clouds for Learning Molecular Properties [55.2480439325792]
  This work introduces AMPTCR, a molecular surface representation that combines local quantum-derived scalar fields and custom topological descriptors within an aligned point cloud format.<n>For molecular weight, results confirm that AMPTCR encodes physically meaningful data, with a validation R2 of 0.87.<n>In the bacterial inhibition task, AMPTCR enables both classification and direct regression of E. coli inhibition values.
  arXiv  Detail & Related papers  (2025-07-22T04:35:50Z)
• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• Molecule Design by Latent Prompt Transformer [76.2112075557233]
  This work explores the challenging problem of molecule design by framing it as a conditional generative modeling task. We propose a novel generative model comprising three components: (1) a latent vector with a learnable prior distribution; (2) a molecule generation model based on a causal Transformer, which uses the latent vector as a prompt; and (3) a property prediction model that predicts a molecule's target properties and/or constraint values using the latent prompt.
  arXiv  Detail & Related papers  (2024-02-27T03:33:23Z)
• GSMFlow: Generation Shifts Mitigating Flow for Generalized Zero-Shot Learning [55.79997930181418]
  Generalized Zero-Shot Learning aims to recognize images from both the seen and unseen classes by transferring semantic knowledge from seen to unseen classes. It is a promising solution to take the advantage of generative models to hallucinate realistic unseen samples based on the knowledge learned from the seen classes. We propose a novel flow-based generative framework that consists of multiple conditional affine coupling layers for learning unseen data generation.
  arXiv  Detail & Related papers  (2022-07-05T04:04:37Z)
• MIMOSA: Multi-constraint Molecule Sampling for Molecule Optimization [51.00815310242277]
  generative models and reinforcement learning approaches made initial success, but still face difficulties in simultaneously optimizing multiple drug properties. We propose the MultI-constraint MOlecule SAmpling (MIMOSA) approach, a sampling framework to use input molecule as an initial guess and sample molecules from the target distribution.
  arXiv  Detail & Related papers  (2020-10-05T20:18:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.