The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification

• URL: http://arxiv.org/abs/2305.17891v2
• Date: Sun, 28 Jan 2024 06:33:33 GMT
• Title: The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification
• Authors: Linhao Qu, Xiaoyuan Luo, Kexue Fu, Manning Wang, Zhijian Song
• Abstract summary: This paper introduces the novel concept of few-shot weakly supervised learning for pathology Whole Slide Image (WSI) classification, denoted as FSWC. A solution is proposed based on prompt learning and the utilization of a large language model, GPT-4. Our approach incorporates the utilization of GPT-4 in a question-and-answer mode to obtain language prior knowledge at both the instance and bag levels, which are then integrated into the instance and bag level language prompts.
• Score: 23.004237397025822
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces the novel concept of few-shot weakly supervised learning for pathology Whole Slide Image (WSI) classification, denoted as FSWC. A solution is proposed based on prompt learning and the utilization of a large language model, GPT-4. Since a WSI is too large and needs to be divided into patches for processing, WSI classification is commonly approached as a Multiple Instance Learning (MIL) problem. In this context, each WSI is considered a bag, and the obtained patches are treated as instances. The objective of FSWC is to classify both bags and instances with only a limited number of labeled bags. Unlike conventional few-shot learning problems, FSWC poses additional challenges due to its weak bag labels within the MIL framework. Drawing inspiration from the recent achievements of vision-language models (V-L models) in downstream few-shot classification tasks, we propose a two-level prompt learning MIL framework tailored for pathology, incorporating language prior knowledge. Specifically, we leverage CLIP to extract instance features for each patch, and introduce a prompt-guided pooling strategy to aggregate these instance features into a bag feature. Subsequently, we employ a small number of labeled bags to facilitate few-shot prompt learning based on the bag features. Our approach incorporates the utilization of GPT-4 in a question-and-answer mode to obtain language prior knowledge at both the instance and bag levels, which are then integrated into the instance and bag level language prompts. Additionally, a learnable component of the language prompts is trained using the available few-shot labeled data. We conduct extensive experiments on three real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer, demonstrating the notable performance of the proposed method in bag and instance classification. All codes will be available.

Related papers

• FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification [23.323845050957196]
  Existing few-shot WSI classification methods only utilize a small number of fine-grained labels or weakly supervised slide labels for training. They lack sufficient mining of available WSIs, severely limiting WSI classification performance. We propose a novel and efficient dual-tier few-shot learning paradigm for WSI classification, named FAST.
  arXiv  Detail & Related papers  (2024-09-29T14:31:52Z)
• MSCPT: Few-shot Whole Slide Image Classification with Multi-scale and Context-focused Prompt Tuning [11.717352903130411]
  Multiple instance learning (MIL) has become a standard paradigm for weakly supervised classification of whole slide images (WSI) The lack of training data and the presence of rare diseases present significant challenges for these methods. We propose a Multi-Scale and Context-focused Prompt Tuning (MSCPT) method for FSWC tasks.
  arXiv  Detail & Related papers  (2024-08-21T10:25:51Z)
• Disambiguated Attention Embedding for Multi-Instance Partial-Label Learning [68.56193228008466]
  In many real-world tasks, the concerned objects can be represented as a multi-instance bag associated with a candidate label set. Existing MIPL approach follows the instance-space paradigm by assigning augmented candidate label sets of bags to each instance and aggregating bag-level labels from instance-level labels. We propose an intuitive algorithm named DEMIPL, i.e., Disambiguated attention Embedding for Multi-Instance Partial-Label learning.
  arXiv  Detail & Related papers  (2023-05-26T13:25:17Z)
• Aligning Bag of Regions for Open-Vocabulary Object Detection [74.89762864838042]
  We propose to align the embedding of bag of regions beyond individual regions. The proposed method groups contextually interrelated regions as a bag. Our approach surpasses the previous best results by 4.6 box AP50 and 2.8 mask AP on novel categories of open-vocabulary COCO and LVIS benchmarks.
  arXiv  Detail & Related papers  (2023-02-27T17:39:21Z)
• Multi-Modal Few-Shot Temporal Action Detection [157.96194484236483]
  Few-shot (FS) and zero-shot (ZS) learning are two different approaches for scaling temporal action detection to new classes. We introduce a new multi-modality few-shot (MMFS) TAD problem, which can be considered as a marriage of FS-TAD and ZS-TAD.
  arXiv  Detail & Related papers  (2022-11-27T18:13:05Z)
• Multiple Instance Learning via Iterative Self-Paced Supervised Contrastive Learning [22.07044031105496]
  Learning representations for individual instances when only bag-level labels are available is a challenge in multiple instance learning (MIL) We propose a novel framework, Iterative Self-paced Supervised Contrastive Learning for MIL Representations (ItS2CLR) It improves the learned representation by exploiting instance-level pseudo labels derived from the bag-level labels.
  arXiv  Detail & Related papers  (2022-10-17T21:43:32Z)
• Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classification [9.43604501642743]
  We propose an end-to-end weakly supervised knowledge distillation framework (WENO) for WSI classification. In this paper, we propose an end-to-end weakly supervised knowledge distillation framework (WENO) for WSI classification.
  arXiv  Detail & Related papers  (2022-10-07T16:12:04Z)
• LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models [67.19124099815645]
  We propose a novel Language-Aware Soft Prompting (LASP) learning method to alleviate base class overfitting. LASP is inherently amenable to including, during training, virtual classes, i.e. class names for which no visual samples are available. LASP matches and surpasses, for the first time, the accuracy on novel classes obtained by hand-crafted prompts and CLIP for 8 out of 11 test datasets.
  arXiv  Detail & Related papers  (2022-10-03T17:56:35Z)
• Multi-Modal Few-Shot Object Detection with Meta-Learning-Based Cross-Modal Prompting [77.69172089359606]
  We study multi-modal few-shot object detection (FSOD) in this paper, using both few-shot visual examples and class semantic information for detection. Our approach is motivated by the high-level conceptual similarity of (metric-based) meta-learning and prompt-based learning. We comprehensively evaluate the proposed multi-modal FSOD models on multiple few-shot object detection benchmarks, achieving promising results.
  arXiv  Detail & Related papers  (2022-04-16T16:45:06Z)
• Accounting for Dependencies in Deep Learning Based Multiple Instance Learning for Whole Slide Imaging [8.712556146101953]
  Multiple instance learning (MIL) is a key algorithm for classification of whole slide images (WSI) Histology WSIs can have billions of pixels, which create enormous computational and annotation challenges. We propose an instance-wise loss function based on instance pseudo-labels.
  arXiv  Detail & Related papers  (2021-11-01T06:50:33Z)
• UniT: Unified Knowledge Transfer for Any-shot Object Detection and Segmentation [52.487469544343305]
  Methods for object detection and segmentation rely on large scale instance-level annotations for training. We propose an intuitive and unified semi-supervised model that is applicable to a range of supervision.
  arXiv  Detail & Related papers  (2020-06-12T22:45:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.