Cross-Modal Prototype Allocation: Unsupervised Slide Representation Learning via Patch-Text Contrast in Computational Pathology

• URL: http://arxiv.org/abs/2503.20190v1
• Date: Wed, 26 Mar 2025 03:31:07 GMT
• Title: Cross-Modal Prototype Allocation: Unsupervised Slide Representation Learning via Patch-Text Contrast in Computational Pathology
• Authors: Yuxuan Chen, Jiawen Li, Jiali Hu, Xitong Ling, Tian Guan, Anjia Han, Yonghong He,
• Abstract summary: ProAlign is a cross-modal unsupervised slide representation learning framework.<n>We leverage a large language model (LLM) to generate descriptive text for the prototype types present in a whole slide image.<n>We propose a parameter-free attention aggregation strategy that utilizes the similarity between patches and these prototypes to form unsupervised slide embeddings.
• Score: 10.811667603360041
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid advancement of pathology foundation models (FMs), the representation learning of whole slide images (WSIs) attracts increasing attention. Existing studies develop high-quality patch feature extractors and employ carefully designed aggregation schemes to derive slide-level representations. However, mainstream weakly supervised slide representation learning methods, primarily based on multiple instance learning (MIL), are tailored to specific downstream tasks, which limits their generalizability. To address this issue, some studies explore unsupervised slide representation learning. However, these approaches focus solely on the visual modality of patches, neglecting the rich semantic information embedded in textual data. In this work, we propose ProAlign, a cross-modal unsupervised slide representation learning framework. Specifically, we leverage a large language model (LLM) to generate descriptive text for the prototype types present in a WSI, introducing patch-text contrast to construct initial prototype embeddings. Furthermore, we propose a parameter-free attention aggregation strategy that utilizes the similarity between patches and these prototypes to form unsupervised slide embeddings applicable to a wide range of downstream tasks. Extensive experiments on four public datasets show that ProAlign outperforms existing unsupervised frameworks and achieves performance comparable to some weakly supervised models.

Related papers

• ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification [52.405499816861635]
  Multiple instance learning (MIL)-based framework has become the mainstream for processing the whole slide image (WSI)<n>We propose a dual-scale vision-language multiple instance learning (ViLa-MIL) framework for whole slide image classification.
  arXiv  Detail & Related papers  (2025-02-12T13:28:46Z)
• Learning Visual Proxy for Compositional Zero-Shot Learning [15.183106475115583]
  We introduce Visual Proxy Learning, a novel approach that facilitates the learning of distinct visual distributions.<n>We propose an effective Cross-Modal Joint Learning strategy that imposes cross-modal constraints between the original text-image space and the fine-grained visual space.
  arXiv  Detail & Related papers  (2025-01-23T17:30:27Z)
• Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP [19.697857943845012]
  We propose a framework to learn class-specific vision prototypes in vision space with the help of text prototypes.<n>We also propose a regional semantic contrast module that contrasts regions embedding with corresponding prototypes.<n>Our proposed framework achieves state-of-the-art performance on two benchmark datasets.
  arXiv  Detail & Related papers  (2024-12-27T13:55:11Z)
• Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning [13.68867780184022]
  Few-shot learning aims to recognize new concepts using a limited number of visual samples. Our framework incorporates both the abstract class semantics and the concrete class entities extracted from Large Language Models (LLMs) For the challenging one-shot setting, our approach, utilizing the ResNet-12 backbone, achieves an average improvement of 1.95% over the second-best competitor.
  arXiv  Detail & Related papers  (2024-08-22T15:10:20Z)
• Correlation Weighted Prototype-based Self-Supervised One-Shot Segmentation of Medical Images [12.365801596593936]
  Medical image segmentation is one of the domains where sufficient annotated data is not available. We propose a prototype-based self-supervised one-way one-shot learning framework using pseudo-labels generated from superpixels. We show that the proposed simple but potent framework performs at par with the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-12T15:38:51Z)
• Beyond Mask: Rethinking Guidance Types in Few-shot Segmentation [67.35274834837064]
  We develop a universal vision-language framework (UniFSS) to integrate prompts from text, mask, box, and image. UniFSS significantly outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-16T08:41:01Z)
• Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control [73.6361029556484]
  Embodied AI agents require a fine-grained understanding of the physical world mediated through visual and language inputs. We consider pre-trained text-to-image diffusion models, which are explicitly optimized to generate images from text prompts. We show that Stable Control Representations enable learning policies that exhibit state-of-the-art performance on OVMM, a difficult open-vocabulary navigation benchmark.
  arXiv  Detail & Related papers  (2024-05-09T15:39:54Z)
• Improving Input-label Mapping with Demonstration Replay for In-context Learning [67.57288926736923]
  In-context learning (ICL) is an emerging capability of large autoregressive language models. We propose a novel ICL method called Sliding Causal Attention (RdSca) We show that our method significantly improves the input-label mapping in ICL demonstrations.
  arXiv  Detail & Related papers  (2023-10-30T14:29:41Z)
• SLPD: Slide-level Prototypical Distillation for WSIs [11.217079419686472]
  We propose Slide-Level Prototypical Distillation (SLPD) to explore intra- and inter-slide semantic structures for context modeling. SLPD achieves state-of-the-art results on multiple slide-level benchmarks and demonstrates that representation learning of semantic structures of slides can make a suitable proxy task for WSI analysis.
  arXiv  Detail & Related papers  (2023-07-20T08:38:15Z)
• Multi-Modal Prototypes for Open-World Semantic Segmentation [37.84805778548119]
  We propose to encompass textual and visual clues as multi-modal prototypes to allow more comprehensive support for semantic segmentation. We decompose the high-level language information as multi-aspect prototypes and aggregate the low-level visual information as more semantic prototypes. Based on an elastic mask prediction module, we are able to solve the zero-shot, few-shot and generalized counterpart tasks in one architecture.
  arXiv  Detail & Related papers  (2023-07-05T03:27:31Z)
• UniT: Unified Knowledge Transfer for Any-shot Object Detection and Segmentation [52.487469544343305]
  Methods for object detection and segmentation rely on large scale instance-level annotations for training. We propose an intuitive and unified semi-supervised model that is applicable to a range of supervision.
  arXiv  Detail & Related papers  (2020-06-12T22:45:47Z)
• Prototypical Contrastive Learning of Unsupervised Representations [171.3046900127166]
  Prototypical Contrastive Learning (PCL) is an unsupervised representation learning method. PCL implicitly encodes semantic structures of the data into the learned embedding space. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks.
  arXiv  Detail & Related papers  (2020-05-11T09:53:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.