M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Images

• URL: http://arxiv.org/abs/2409.15092v4
• Date: Fri, 20 Dec 2024 06:19:10 GMT
• Title: M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Images
• Authors: Hongyi Wang, Xiuju Du, Jing Liu, Shuyi Ouyang, Yen-Wei Chen, Lanfen Lin,
• Abstract summary: We propose M2OST, a many-to-one regression Transformer that can accommodate the hierarchical structure of pathology images. Unlike traditional models that are trained with one-to-one image-label pairs, M2OST uses multiple images from different levels of the digital pathology image to jointly predict the gene expressions in their common corresponding spot. M2OST can achieve state-of-the-art performance with fewer parameters and floating-point operations (FLOPs)
• Score: 16.19308597273405
• License:
• Abstract: The advancement of Spatial Transcriptomics (ST) has facilitated the spatially-aware profiling of gene expressions based on histopathology images. Although ST data offers valuable insights into the micro-environment of tumors, its acquisition cost remains expensive. Therefore, directly predicting the ST expressions from digital pathology images is desired. Current methods usually adopt existing regression backbones along with patch-sampling for this task, which ignores the inherent multi-scale information embedded in the pyramidal data structure of digital pathology images, and wastes the inter-spot visual information crucial for accurate gene expression prediction. To address these limitations, we propose M2OST, a many-to-one regression Transformer that can accommodate the hierarchical structure of the pathology images via a decoupled multi-scale feature extractor. Unlike traditional models that are trained with one-to-one image-label pairs, M2OST uses multiple images from different levels of the digital pathology image to jointly predict the gene expressions in their common corresponding spot. Built upon our many-to-one scheme, M2OST can be easily scaled to fit different numbers of inputs, and its network structure inherently incorporates nearby inter-spot features, enhancing regression performance. We have tested M2OST on three public ST datasets and the experimental results show that M2OST can achieve state-of-the-art performance with fewer parameters and floating-point operations (FLOPs).

Related papers

• M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Images [17.158450092707042]
  M2ORT is a many-to-one regression Transformer that can accommodate the hierarchical structure of pathology images. We have tested M2ORT on three public ST datasets and the experimental results show that M2ORT can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-01-19T10:37:27Z)
• Enhancing CT Image synthesis from multi-modal MRI data based on a multi-task neural network framework [16.864720020158906]
  We propose a versatile multi-task neural network framework, based on an enhanced Transformer U-Net architecture. We decompose the traditional problem of synthesizing CT images into distinct subtasks. To enhance the framework's versatility in handling multi-modal data, we expand the model with multiple image channels.
  arXiv  Detail & Related papers  (2023-12-13T18:22:38Z)
• Masked Pre-Training of Transformers for Histology Image Analysis [4.710921988115685]
  In digital pathology, whole slide images (WSIs) are widely used for applications such as cancer diagnosis and prognosis prediction. Visual transformer models have emerged as a promising method for encoding large regions of WSIs while preserving spatial relationships among patches. We propose a pretext task for training the transformer model without labeled data to address this problem. Our model, MaskHIT, uses the transformer output to reconstruct masked patches and learn representative histological features based on their positions and visual features.
  arXiv  Detail & Related papers  (2023-04-14T23:56:49Z)
• PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysis [56.63327669853693]
  We propose to incorporate the task of pixel restoration for explicitly encoding more pixel-level information into high-level semantics. We also address the preservation of scale information, a powerful tool in aiding image understanding. The proposed unified SSL framework surpasses its self-supervised counterparts on various tasks.
  arXiv  Detail & Related papers  (2023-01-02T17:47:27Z)
• Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics [63.76637479503006]
  Learning good representation of giga-pixel level whole slide pathology images (WSI) for downstream tasks is critical. This paper proposes a hierarchical-based multimodal transformer framework that learns a hierarchical mapping between pathology images and corresponding genes. Our architecture requires fewer GPU resources compared with benchmark methods while maintaining better WSI representation ability.
  arXiv  Detail & Related papers  (2022-11-29T23:47:56Z)
• AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation [50.21065317817769]
  We propose an AlignTransformer framework, which includes the Align Hierarchical Attention (AHA) and the Multi-Grained Transformer (MGT) modules. Experiments on the public IU-Xray and MIMIC-CXR datasets show that the AlignTransformer can achieve results competitive with state-of-the-art methods on the two datasets.
  arXiv  Detail & Related papers  (2022-03-18T13:43:53Z)
• Class-Aware Generative Adversarial Transformers for Medical Image Segmentation [39.14169989603906]
  We present CA-GANformer, a novel type of generative adversarial transformers, for medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design a novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures.
  arXiv  Detail & Related papers  (2022-01-26T03:50:02Z)
• Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation [75.58395328700821]
  We propose a novel model, Multi-modal Gaussian Process Prior Variational Autoencoder (MGP-VAE), to impute one or more missing sub-modalities for a patient scan. MGP-VAE can leverage the Gaussian Process (GP) prior on the Variational Autoencoder (VAE) to utilize the subjects/patients and sub-modalities correlations. We show the applicability of MGP-VAE on brain tumor segmentation where either, two, or three of four sub-modalities may be missing.
  arXiv  Detail & Related papers  (2021-07-07T19:06:34Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)
• Image-level Harmonization of Multi-Site Data using Image-and-Spatial Transformer Networks [0.0]
  We use image-and-spatial transformer networks (ISTNs) to tackle domain shift in medical imaging data. We employ ISTNs for domain adaptation at the image-level which constrains transformations to explainable appearance and shape changes. For real-data validation, we construct two 3D brain MRI datasets from the Cam-CAN and UK Biobank studies.
  arXiv  Detail & Related papers  (2020-06-30T12:58:41Z)
• MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data [75.73881040581767]
  We propose a novel multi-site network (MS-Net) for improving prostate segmentation by learning robust representations. Our MS-Net improves the performance across all datasets consistently, and outperforms state-of-the-art methods for multi-site learning.
  arXiv  Detail & Related papers  (2020-02-09T14:11:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.