Fluoroformer: Scaling multiple instance learning to multiplexed images via attention-based channel fusion

• URL: http://arxiv.org/abs/2411.08975v1
• Date: Wed, 13 Nov 2024 19:06:57 GMT
• Title: Fluoroformer: Scaling multiple instance learning to multiplexed images via attention-based channel fusion
• Authors: Marc Harary, Eliezer M. Van Allen, William Lotter,
• Abstract summary: We present an MIL strategy, the Fluoroformer module, that is specifically tailored to multiplexed whole slide images. On a cohort of 434 non-small cell lung cancer (NSCLC) samples, we show that the Fluoroformer both obtains strong prognostic performance and recapitulates immuno-oncological hallmarks.
• Score: 0.03511246202322249
• License:
• Abstract: Though multiple instance learning (MIL) has been a foundational strategy in computational pathology for processing whole slide images (WSIs), current approaches are designed for traditional hematoxylin and eosin (H&E) slides rather than emerging multiplexed technologies. Here, we present an MIL strategy, the Fluoroformer module, that is specifically tailored to multiplexed WSIs by leveraging scaled dot-product attention (SDPA) to interpretably fuse information across disparate channels. On a cohort of 434 non-small cell lung cancer (NSCLC) samples, we show that the Fluoroformer both obtains strong prognostic performance and recapitulates immuno-oncological hallmarks of NSCLC. Our technique thereby provides a path for adapting state-of-the-art AI techniques to emerging spatial biology assays.

Related papers

• CARMIL: Context-Aware Regularization on Multiple Instance Learning models for Whole Slide Images [0.41873161228906586]
  Multiple Instance Learning models have proven effective for cancer prognosis from Whole Slide Images. The original MIL formulation incorrectly assumes the patches of the same image to be independent. We propose a versatile regularization scheme designed to seamlessly integrate spatial knowledge into any MIL model.
  arXiv  Detail & Related papers  (2024-08-01T09:59:57Z)
• An efficient framework based on large foundation model for cervical cytopathology whole slide image screening [13.744580492120749]
  We propose an efficient framework for cervical cytopathology WSI classification using only WSI-level labels through unsupervised and weakly supervised learning. Experiments conducted on the CSD and FNAC 2019 datasets demonstrate that the proposed method enhances the performance of various MIL methods and achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2024-07-16T08:21:54Z)
• Enhancing Weakly-Supervised Histopathology Image Segmentation with Knowledge Distillation on MIL-Based Pseudo-Labels [8.934328206473456]
  We propose a novel distillation framework for histopathology image segmentation. This framework introduces a iterative fusion-knowledge distillation strategy, enabling the student model to learn directly from the teacher's comprehensive outcomes.
  arXiv  Detail & Related papers  (2024-07-14T17:15:47Z)
• Affine-Consistent Transformer for Multi-Class Cell Nuclei Detection [76.11864242047074]
  We propose a novel Affine-Consistent Transformer (AC-Former), which directly yields a sequence of nucleus positions. We introduce an Adaptive Affine Transformer (AAT) module, which can automatically learn the key spatial transformations to warp original images for local network training. Experimental results demonstrate that the proposed method significantly outperforms existing state-of-the-art algorithms on various benchmarks.
  arXiv  Detail & Related papers  (2023-10-22T02:27:02Z)
• The Whole Pathological Slide Classification via Weakly Supervised Learning [7.313528558452559]
  We introduce two pathological priors: nuclear disease of cells and spatial correlation of pathological tiles. We propose a data augmentation method that utilizes stain separation during extractor training. We then describe the spatial relationships between the tiles using an adjacency matrix. By integrating these two views, we designed a multi-instance framework for analyzing H&E-stained tissue images.
  arXiv  Detail & Related papers  (2023-07-12T16:14:23Z)
• Active Learning Enhances Classification of Histopathology Whole Slide Images with Attention-based Multiple Instance Learning [48.02011627390706]
  We train an attention-based MIL and calculate a confidence metric for every image in the dataset to select the most uncertain WSIs for expert annotation. With a novel attention guiding loss, this leads to an accuracy boost of the trained models with few regions annotated for each class. It may in the future serve as an important contribution to train MIL models in the clinically relevant context of cancer classification in histopathology.
  arXiv  Detail & Related papers  (2023-03-02T15:18:58Z)
• AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images [53.29794593104923]
  We present a novel concept of shared-context processing for whole slide histopathology images. AMIGO uses the celluar graph within the tissue to provide a single representation for a patient. We show that our model is strongly robust to missing information to an extent that it can achieve the same performance with as low as 20% of the data.
  arXiv  Detail & Related papers  (2023-03-01T23:37:45Z)
• Orientation-Shared Convolution Representation for CT Metal Artifact Learning [63.67718355820655]
  During X-ray computed tomography (CT) scanning, metallic implants carrying with patients often lead to adverse artifacts. Existing deep-learning-based methods have gained promising reconstruction performance. We propose an orientation-shared convolution representation strategy to adapt the physical prior structures of artifacts.
  arXiv  Detail & Related papers  (2022-12-26T13:56:12Z)
• Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics [63.76637479503006]
  Learning good representation of giga-pixel level whole slide pathology images (WSI) for downstream tasks is critical. This paper proposes a hierarchical-based multimodal transformer framework that learns a hierarchical mapping between pathology images and corresponding genes. Our architecture requires fewer GPU resources compared with benchmark methods while maintaining better WSI representation ability.
  arXiv  Detail & Related papers  (2022-11-29T23:47:56Z)
• Cross-scale Attention Guided Multi-instance Learning for Crohn's Disease Diagnosis with Pathological Images [22.98849180654734]
  Multi-instance learning (MIL) is widely used in the computer-aided interpretation of pathological Whole Slide Images (WSIs) We propose a novel cross-scale attention mechanism to explicitly aggregate inter-scale interactions into a single MIL network for Crohn's Disease (CD) Our approach achieved a superior Area under the Curve (AUC) score of 0.8924 compared with baseline models.
  arXiv  Detail & Related papers  (2022-08-15T16:39:34Z)
• Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction [138.04956118993934]
  We propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST) CST embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing.
  arXiv  Detail & Related papers  (2022-03-09T16:17:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.