GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathology

• URL: http://arxiv.org/abs/2504.01009v1
• Date: Tue, 01 Apr 2025 17:49:59 GMT
• Title: GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathology
• Authors: Saarthak Kapse, Pushpak Pati, Srikar Yellapragada, Srijan Das, Rajarsi R. Gupta, Joel Saltz, Dimitris Samaras, Prateek Prasanna,
• Abstract summary: We propose Gigapixel Vision-Concept Knowledge Contrastive pretraining.<n>It aligns Whole Slide Image (WSI)-level embeddings with a Concept Prior.<n>It consistently outperforms prior unimodal and multimodal pretraining approaches.
• Score: 26.766788349092476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretraining a Multiple Instance Learning (MIL) aggregator enables the derivation of Whole Slide Image (WSI)-level embeddings from patch-level representations without supervision. While recent multimodal MIL pretraining approaches leveraging auxiliary modalities have demonstrated performance gains over unimodal WSI pretraining, the acquisition of these additional modalities necessitates extensive clinical profiling. This requirement increases costs and limits scalability in existing WSI datasets lacking such paired modalities. To address this, we propose Gigapixel Vision-Concept Knowledge Contrastive pretraining (GECKO), which aligns WSIs with a Concept Prior derived from the available WSIs. First, we derive an inherently interpretable concept prior by computing the similarity between each WSI patch and textual descriptions of predefined pathology concepts. GECKO then employs a dual-branch MIL network: one branch aggregates patch embeddings into a WSI-level deep embedding, while the other aggregates the concept prior into a corresponding WSI-level concept embedding. Both aggregated embeddings are aligned using a contrastive objective, thereby pretraining the entire dual-branch MIL model. Moreover, when auxiliary modalities such as transcriptomics data are available, GECKO seamlessly integrates them. Across five diverse tasks, GECKO consistently outperforms prior unimodal and multimodal pretraining approaches while also delivering clinically meaningful interpretability that bridges the gap between computational models and pathology expertise. Code is made available at https://github.com/bmi-imaginelab/GECKO

Related papers

• HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction [55.00788339683146]
  We propose a novel Hierarchical vision-Language collaboration framework for improved survival prediction.<n> Specifically, HiLa employs pretrained feature extractors to generate hierarchical visual features from WSIs at both patch and region levels.<n>This ap-proach enables the comprehensive learning of discriminative visual features cor-responding to different survival-related attributes from prompts.
  arXiv  Detail & Related papers  (2025-07-07T02:06:25Z)
• SimMIL: A Universal Weakly Supervised Pre-Training Framework for Multi-Instance Learning in Whole Slide Pathology Images [12.827931905880163]
  This paper proposes to pre-train feature extractor for MIL via a weakly-supervised scheme.<n>To learn effective features for MIL, we delve into several key components, including strong data augmentation, a non-linear prediction head and the robust loss function.<n>We conduct experiments on common large-scale WSI datasets and find it achieves better performance than other pre-training schemes.
  arXiv  Detail & Related papers  (2025-05-10T17:23:36Z)
• Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognition [86.21199607040147]
  Self-Improving cognition (SIcog) is a self-learning framework for constructing next-generation foundation language models.<n>We introduce Chain-of-Description, a step-by-step visual understanding method, and integrate structured chain-of-thought (CoT) reasoning to support in-depth multimodal reasoning.<n>Extensive experiments demonstrate that SIcog produces next-generation foundation MLLMs with substantially improved multimodal cognition.
  arXiv  Detail & Related papers  (2025-03-16T00:25:13Z)
• Aggregation Schemes for Single-Vector WSI Representation Learning in Digital Pathology [2.0088541799100392]
  A crucial step to efficiently integrate Whole Slide Images (WSIs) in computational pathology is assigning a single high-quality feature vector, i.e., one embedding, to each WSI.<n>In this paper, we evaluate the WSI search performance of multiple recently developed aggregation techniques.
  arXiv  Detail & Related papers  (2025-01-29T18:14:51Z)
• Revisiting the Integration of Convolution and Attention for Vision Backbone [59.50256661158862]
  Convolutions and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. We propose in this work to use MSHAs and Convs in parallel textbfat different granularity levels instead. We empirically verify the potential of the proposed integration scheme, named textitGLMix: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few semantic slots.
  arXiv  Detail & Related papers  (2024-11-21T18:59:08Z)
• Queryable Prototype Multiple Instance Learning with Vision-Language Models for Incremental Whole Slide Image Classification [10.667645628712542]
  Whole Slide Image (WSI) classification has very significant applications in clinical pathology.<n>This paper proposes the first Vision-Language-based framework with Queryable Prototype Multiple Instance Learning (QPMIL-VL) specially designed for incremental WSI classification.
  arXiv  Detail & Related papers  (2024-10-14T14:49:34Z)
• MSCPT: Few-shot Whole Slide Image Classification with Multi-scale and Context-focused Prompt Tuning [11.717352903130411]
  Multiple instance learning (MIL) has become a standard paradigm for weakly supervised classification of whole slide images (WSI) The lack of training data and the presence of rare diseases present significant challenges for these methods. We propose a Multi-Scale and Context-focused Prompt Tuning (MSCPT) method for FSWC tasks.
  arXiv  Detail & Related papers  (2024-08-21T10:25:51Z)
• ULTRA-DP: Unifying Graph Pre-training with Multi-task Graph Dual Prompt [67.8934749027315]
  We propose a unified framework for graph hybrid pre-training which injects the task identification and position identification into GNNs. We also propose a novel pre-training paradigm based on a group of $k$-nearest neighbors.
  arXiv  Detail & Related papers  (2023-10-23T12:11:13Z)
• ComPtr: Towards Diverse Bi-source Dense Prediction Tasks via A Simple yet General Complementary Transformer [91.43066633305662]
  We propose a novel underlineComPlementary underlinetransformer, textbfComPtr, for diverse bi-source dense prediction tasks. ComPtr treats different inputs equally and builds an efficient dense interaction model in the form of sequence-to-sequence on top of the transformer.
  arXiv  Detail & Related papers  (2023-07-23T15:17:45Z)
• Dual-Query Multiple Instance Learning for Dynamic Meta-Embedding based Tumor Classification [5.121989578393729]
  Whole slide image (WSI) assessment is a challenging and crucial step in cancer diagnosis and treatment planning. Coarse-grained labels are easily accessible, which makes WSI classification an ideal use case for multiple instance learning (MIL) We propose a novel embedding-based Dual-Query MIL pipeline (DQ-MIL)
  arXiv  Detail & Related papers  (2023-07-14T17:06:49Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information [77.80071279597665]
  We propose an all-in-one single-stage pre-training approach, named Maximizing Multi-modal Mutual Information Pre-training (M3I Pre-training) Our approach achieves better performance than previous pre-training methods on various vision benchmarks, including ImageNet classification, object detection, LVIS long-tailed object detection, and ADE20k semantic segmentation.
  arXiv  Detail & Related papers  (2022-11-17T18:59:49Z)
• Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training [88.80694147730883]
  We investigate a variety of Modality-Shared Contrastive Language-Image Pre-training (MS-CLIP) frameworks. In studied conditions, we observe that a mostly unified encoder for vision and language signals outperforms all other variations that separate more parameters. Our approach outperforms vanilla CLIP by 1.6 points in linear probing on a collection of 24 downstream vision tasks.
  arXiv  Detail & Related papers  (2022-07-26T05:19:16Z)
• Cluster-to-Conquer: A Framework for End-to-End Multi-Instance Learning for Whole Slide Image Classification [7.876654642325896]
  We propose an end-to-end framework that clusters the patches from a Whole Slide Images (WSI) into $k$-groups, samples $k'$ patches from each group for training, and uses an adaptive attention mechanism for slide level prediction. The framework is optimized end-to-end on slide-level cross-entropy, patch-level cross-entropy, and KL-divergence loss.
  arXiv  Detail & Related papers  (2021-03-19T04:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.