MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models

• URL: http://arxiv.org/abs/2505.10294v1
• Date: Thu, 15 May 2025 13:42:48 GMT
• Title: MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models
• Authors: Guillaume Balezo, Roger Trullo, Albert Pla Planas, Etienne Decenciere, Thomas Walter,
• Abstract summary: We introduce MIPHEI, a U-Net-inspired architecture that integrates state-of-the-art ViT foundation models as encoders to predict mIF signals from H&E images.<n>We train our model using the publicly available ORION dataset of restained H&E and mIF images from colorectal cancer tissue.<n>MIPHEI achieves accurate cell-type classification from H&E alone, with F1 scores of 0.88 for Pan-CK, 0.57 for CD3e, 0.56 for SMA, 0.36 for CD68, and 0.30 for CD20.
• Score: 2.2802322828460864
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Histopathological analysis is a cornerstone of cancer diagnosis, with Hematoxylin and Eosin (H&E) staining routinely acquired for every patient to visualize cell morphology and tissue architecture. On the other hand, multiplex immunofluorescence (mIF) enables more precise cell type identification via proteomic markers, but has yet to achieve widespread clinical adoption due to cost and logistical constraints. To bridge this gap, we introduce MIPHEI (Multiplex Immunofluorescence Prediction from H&E), a U-Net-inspired architecture that integrates state-of-the-art ViT foundation models as encoders to predict mIF signals from H&E images. MIPHEI targets a comprehensive panel of markers spanning nuclear content, immune lineages (T cells, B cells, myeloid), epithelium, stroma, vasculature, and proliferation. We train our model using the publicly available ORION dataset of restained H&E and mIF images from colorectal cancer tissue, and validate it on two independent datasets. MIPHEI achieves accurate cell-type classification from H&E alone, with F1 scores of 0.88 for Pan-CK, 0.57 for CD3e, 0.56 for SMA, 0.36 for CD68, and 0.30 for CD20, substantially outperforming both a state-of-the-art baseline and a random classifier for most markers. Our results indicate that our model effectively captures the complex relationships between nuclear morphologies in their tissue context, as visible in H&E images and molecular markers defining specific cell types. MIPHEI offers a promising step toward enabling cell-type-aware analysis of large-scale H&E datasets, in view of uncovering relationships between spatial cellular organization and patient outcomes.

Related papers

• Detecting immune cells with label-free two-photon autofluorescence and deep learning [2.096497520824982]
  We trained a convolutional neural network (CNN) to classify cell types based on this label-free AF as input.<n>A low-complexity squeezeNet architecture was able to achieve reliable immune cell classification results.<n>In the future, such predictive DL models could directly detect specific immune cells in unstained images.
  arXiv  Detail & Related papers  (2025-06-17T12:14:02Z)
• Mew: Multiplexed Immunofluorescence Image Analysis through an Efficient Multiplex Network [84.88767228835928]
  We introduce Mew, a novel framework designed to efficiently process mIF images through the lens of multiplex network. Mew innovatively constructs a multiplex network comprising two distinct layers: a Voronoi network for geometric information and a Cell-type network for capturing cell-wise homogeneity. This framework equips a scalable and efficient Graph Neural Network (GNN), capable of processing the entire graph during training.
  arXiv  Detail & Related papers  (2024-07-25T08:22:30Z)
• Semi-supervised variational autoencoder for cell feature extraction in multiplexed immunofluorescence images [40.234346302444536]
  We propose a deep learning-based cell feature extraction model using a variational autoencoder with supervision. We perform cell phenotype classification using a cohort of more than 44,000 multiplexed immunofluorescence cell image patches extracted across 1,093 tissue microarray cores of breast cancer patients.
  arXiv  Detail & Related papers  (2024-06-22T04:32:50Z)
• MMIL: A novel algorithm for disease associated cell type discovery [58.044870442206914]
  Single-cell datasets often lack individual cell labels, making it challenging to identify cells associated with disease. We introduce Mixture Modeling for Multiple Learning Instance (MMIL), an expectation method that enables the training and calibration of cell-level classifiers.
  arXiv  Detail & Related papers  (2024-06-12T15:22:56Z)
• Immunocto: a massive immune cell database auto-generated for histopathology [0.0]
  We create Immunocto, a massive, multi-million automatically generated database of 6,848,454 human cells and objects.<n>For each cell, we provide a 64$times$64 pixels$2$ H&E image at $mathbf40times$ magnification, along with a binary mask of the nucleus and a label.<n>Deep learning models trained on Immunocto result in state-of-the-art performance for lymphocyte detection.
  arXiv  Detail & Related papers  (2024-06-03T17:03:58Z)
• UniCell: Universal Cell Nucleus Classification via Prompt Learning [76.11864242047074]
  We propose a universal cell nucleus classification framework (UniCell) It employs a novel prompt learning mechanism to uniformly predict the corresponding categories of pathological images from different dataset domains. In particular, our framework adopts an end-to-end architecture for nuclei detection and classification, and utilizes flexible prediction heads for adapting various datasets.
  arXiv  Detail & Related papers  (2024-02-20T11:50:27Z)
• Mixed Models with Multiple Instance Learning [51.440557223100164]
  We introduce MixMIL, a framework integrating Generalized Linear Mixed Models (GLMM) and Multiple Instance Learning (MIL) Our empirical results reveal that MixMIL outperforms existing MIL models in single-cell datasets.
  arXiv  Detail & Related papers  (2023-11-04T16:42:42Z)
• Structural Cycle GAN for Virtual Immunohistochemistry Staining of Gland Markers in the Colon [1.741980945827445]
  Hematoxylin and Eosin (H&E) staining is one of the most frequently used stains for disease analysis, diagnosis, and grading. Pathologists do need differentchemical (IHC) stains to analyze specific structures or cells. Hematoxylin and Eosin (H&E) staining is one of the most frequently used stains for disease analysis, diagnosis, and grading.
  arXiv  Detail & Related papers  (2023-08-25T05:24:23Z)
• AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images [53.29794593104923]
  We present a novel concept of shared-context processing for whole slide histopathology images. AMIGO uses the celluar graph within the tissue to provide a single representation for a patient. We show that our model is strongly robust to missing information to an extent that it can achieve the same performance with as low as 20% of the data.
  arXiv  Detail & Related papers  (2023-03-01T23:37:45Z)
• Three-dimensional micro-structurally informed in silico myocardium -- towards virtual imaging trials in cardiac diffusion weighted MRI [58.484353709077034]
  We propose a novel method to generate a realistic numerical phantom of myocardial microstructure. In-silico tissue models enable evaluating quantitative models of magnetic resonance imaging.
  arXiv  Detail & Related papers  (2022-08-22T22:01:44Z)
• Automated Phenotyping via Cell Auto Training (CAT) on the Cell DIVE Platform [0.5599792629509229]
  We present a method for automatic cell classification in tissue samples using an automated training set from multiplexed immunofluorescence images. The method utilizes multiple markers stained in situ on a single tissue section on a robust hyperplex immunofluorescence platform.
  arXiv  Detail & Related papers  (2020-07-18T16:45:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.