CytoFormer: A Molecularly Supervised Cell Foundation Model for Histopathology Cell Classification

• URL: http://arxiv.org/abs/2608.16718v1
• Date: Mon, 17 Aug 2026 15:35:03 GMT
• Title: CytoFormer: A Molecularly Supervised Cell Foundation Model for Histopathology Cell Classification
• Abstract summary: Training such models has relied on manual pathologist annotations, which are slow, expensive and unreliable for many cell types.<n>We instead supervise morphology with molecules.<n>We assembled 81 such paired Xenium sections spanning 16 organs, derived per-cell labels by clustering, marker-gene annotation, organ-wise human review and quality control, and mapped them onto the cell types commonly reported in each organ.<n>This yielded 15.4 million cells, each with a paired H&E image patch and one of 23 cell types, on which we trained CytoFormer, a cell foundation model with a multi-task, per-organ classification head.
• Score: 1.5928576802540013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Identifying cell types directly from routine haematoxylin and eosin (H&E) histology would enable single-cell analysis at scale, but training such models has relied on manual pathologist annotations, which are slow, expensive and unreliable for many cell types. We instead supervise morphology with molecules. Imaging-based spatial transcriptomics profiles individual cells in situ on a section that can afterwards be stained with H&E, so that molecular identity and morphology are observed for the same physical cell. We assembled 81 such paired Xenium sections spanning 16 organs, derived per-cell labels by clustering, marker-gene annotation, organ-wise human review and quality control, and mapped them onto the cell types commonly reported in each organ. This yielded 15.4 million cells, each with a paired H&E image patch and one of 23 cell types, on which we trained CytoFormer, a cell foundation model with a multi-task, per-organ classification head. On spatially held-out tissue CytoFormer reached an accuracy of 0.85 and a macro-F1 of 0.78 across all 16 organs, and its predictions reproduced the tissue architecture of an entire held-out section. The representation also transfers: with the encoder frozen, a linear head on CytoFormer features performed better than six pathology foundation models on four expert-annotated benchmarks, including on organs and cell types that were not part of pretraining. Finally, in an interactive active-learning setting, CytoFormer's embeddings are markedly more label-efficient than existing pathology foundation models, detecting normal epithelium amid look-alike tumour with an F1 of 0.82 from only a few annotations and leading the strongest baseline by 0.13 in F1. CytoFormer turns paired H&E and spatial transcriptomics into a reusable, label-efficient representation for cell-level analysis of routine histology.

Related papers

• Lingshu-Cell: A generative cellular world model for transcriptome modeling toward virtual cells [48.885716676667776]
  Lingshu-Cell is a discrete diffusion model that learns transcriptomic state distributions and supports conditional simulation under perturbations.<n>It captures complex transcriptome-wide expression dependencies across approximately 18,000 genes without relying on prior gene selection.<n>It achieves leading performance on the Virtual Cell Challenge H1 genetic perturbation benchmark and in predicting cytokine-induced responses in human PBMCs.
  arXiv  Detail & Related papers  (2026-03-26T09:46:27Z)
• Learning Cell-Aware Hierarchical Multi-Modal Representations for Robust Molecular Modeling [74.25438319700929]
  We propose CHMR (Cell-aware Hierarchical Multi-modal Representations), a robust framework that models local-global dependencies between molecules and cellular responses.<n> evaluated on nine public benchmarks spanning 728 tasks, CHMR outperforms state-of-the-art baselines.<n>Results demonstrate the advantage of hierarchy-aware, multimodal learning for reliable and biologically grounded molecular representations.
  arXiv  Detail & Related papers  (2025-11-26T07:15:00Z)
• From Linear Probing to Joint-Weighted Token Hierarchy: A Foundation Model Bridging Global and Cellular Representations in Biomarker Detection [44.3895875409365]
  AI-based biomarkers can infer molecular features directly from hematoxylin & eosin (H&E) slides.<n>Most pathology foundation models (PFMs) rely on global patch-level embeddings and overlook cell-level morphology.<n>We present a PFM model, JWTH, which integrates large-scale self-supervised pretraining with cell-centric post-tuning and attention pooling to fuse local and global tokens.
  arXiv  Detail & Related papers  (2025-11-07T11:05:36Z)
• Automatic Classification of Circulating Blood Cell Clusters based on Multi-channel Flow Cytometry Imaging [7.696386233227689]
  Circulating blood cell clusters (CCCs) containing red blood cells (RBCs), white blood cells(WBCs) and platelets are significant biomarkers linked to conditions like thrombosis, infection, and inflammation.<n> Flow, paired with fluorescence staining, is commonly used to analyze these cell clusters, revealing cell morphology and protein profiles.<n>This study introduces a new computational framework for analyzing CCC images and identifying cell types within clusters.
  arXiv  Detail & Related papers  (2025-10-20T16:32:23Z)
• Towards Comprehensive Cellular Characterisation of H&E slides [0.27993409178463413]
  HistoPLUS is a state-of-the-art model for cell analysis.<n>It trains on a novel curated pan-cancer dataset of 108,722 nuclei covering 13 cell types.<n>It outperforms current state-of-the-art models in detection quality by 5.2% and overall F1 classification score by 23.7%.
  arXiv  Detail & Related papers  (2025-08-13T16:24:15Z)
• MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models [2.2802322828460864]
  We introduce MIPHEI, a U-Net-inspired architecture that integrates state-of-the-art ViT foundation models as encoders to predict mIF signals from H&E images.<n>We train our model using the publicly available ORION dataset of restained H&E and mIF images from colorectal cancer tissue.<n>MIPHEI achieves accurate cell-type classification from H&E alone, with F1 scores of 0.88 for Pan-CK, 0.57 for CD3e, 0.56 for SMA, 0.36 for CD68, and 0.30 for CD20.
  arXiv  Detail & Related papers  (2025-05-15T13:42:48Z)
• CellVerse: Do Large Language Models Really Understand Cell Biology? [74.34984441715517]
  We introduce CellVerse, a unified language-centric question-answering benchmark that integrates four types of single-cell multi-omics data.<n>We systematically evaluate the performance across 14 open-source and closed-source LLMs ranging from 160M to 671B on CellVerse.
  arXiv  Detail & Related papers  (2025-05-09T06:47:23Z)
• Immunohistochemistry guided segmentation of benign epithelial cells, in situ lesions, and invasive epithelial cells in breast cancer slides [0.3251634769699391]
  We developed an AI model for segmentation of epithelial cells in sections from breast cancer. Quantitative evaluation, a mean Dice score of 0.70, 0.79, and 0.75 for invasive epithelial cells, benign epithelial cells, and in situ lesions, respectively, were achieved.
  arXiv  Detail & Related papers  (2023-11-22T09:25:08Z)
• Multi-stream Cell Segmentation with Low-level Cues for Multi-modality Images [66.79688768141814]
  We develop an automatic cell classification pipeline to label microscopy images. We then train a classification model based on the category labels. We deploy two types of segmentation models to segment cells with roundish and irregular shapes.
  arXiv  Detail & Related papers  (2023-10-22T08:11:08Z)
• Multilabel 12-Lead Electrocardiogram Classification Using Gradient Boosting Tree Ensemble [64.29529357862955]
  We build an algorithm using gradient boosted tree ensembles fitted on morphology and signal processing features to classify ECG diagnosis. For each lead, we derive features from heart rate variability, PQRST template shape, and the full signal waveform. We join the features of all 12 leads to fit an ensemble of gradient boosting decision trees to predict probabilities of ECG instances belonging to each class.
  arXiv  Detail & Related papers  (2020-10-21T18:11:36Z)
• Cell Mechanics Based Computational Classification of Red Blood Cells Via Machine Intelligence Applied to Morpho-Rheological Markers [0.0]
  Unsupervised machine learning methodology is applied exclusively on morpho-rheological markers obtained by real-time deformability and fluorescence (RT-FDC) Our approach reports promising label-free results in the classification of reticulocytes from mature red blood cells.
  arXiv  Detail & Related papers  (2020-03-02T15:11:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.