How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark

• URL: http://arxiv.org/abs/2503.12990v1
• Date: Mon, 17 Mar 2025 09:45:22 GMT
• Title: How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark
• Authors: Roba Al Majzoub, Hashmat Malik, Muzammal Naseer, Zaigham Zaheer, Tariq Mahmood, Salman Khan, Fahad Khan,
• Abstract summary: Histopathology vision-language foundation models (VLMs) have gained popularity due to their enhanced performance and generalizability across different downstream tasks.<n>Most existing histopathology benchmarks are either unimodal or limited in terms of diversity of clinical tasks, organs, and acquisition instruments, as well as their partial availability to the public due to patient data privacy.<n>We introduce HistoVL, a fully open-source comprehensive benchmark comprising images acquired using up to 11 various acquisition tools and captions by incorporating class names and diverse pathology descriptions.
• Score: 21.47220651857942
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, histopathology vision-language foundation models (VLMs) have gained popularity due to their enhanced performance and generalizability across different downstream tasks. However, most existing histopathology benchmarks are either unimodal or limited in terms of diversity of clinical tasks, organs, and acquisition instruments, as well as their partial availability to the public due to patient data privacy. As a consequence, there is a lack of comprehensive evaluation of existing histopathology VLMs on a unified benchmark setting that better reflects a wide range of clinical scenarios. To address this gap, we introduce HistoVL, a fully open-source comprehensive benchmark comprising images acquired using up to 11 various acquisition tools that are paired with specifically crafted captions by incorporating class names and diverse pathology descriptions. Our Histo-VL includes 26 organs, 31 cancer types, and a wide variety of tissue obtained from 14 heterogeneous patient cohorts, totaling more than 5 million patches obtained from over 41K WSIs viewed under various magnification levels. We systematically evaluate existing histopathology VLMs on Histo-VL to simulate diverse tasks performed by experts in real-world clinical scenarios. Our analysis reveals interesting findings, including large sensitivity of most existing histopathology VLMs to textual changes with a drop in balanced accuracy of up to 25% in tasks such as Metastasis detection, low robustness to adversarial attacks, as well as improper calibration of models evident through high ECE values and low model prediction confidence, all of which can affect their clinical implementation.

Related papers

• PathOrchestra: A Comprehensive Foundation Model for Computational Pathology with Over 100 Diverse Clinical-Grade Tasks [39.97710183184273]
  We present PathOrchestra, a versatile pathology foundation model trained via self-supervised learning on a dataset comprising 300K pathological slides. The model was rigorously evaluated on 112 clinical tasks using a combination of 61 private and 51 public datasets. PathOrchestra demonstrated exceptional performance across 27,755 WSIs and 9,415,729 ROIs, achieving over 0.950 accuracy in 47 tasks.
  arXiv  Detail & Related papers  (2025-03-31T17:28:02Z)
• MIL vs. Aggregation: Evaluating Patient-Level Survival Prediction Strategies Using Graph-Based Learning [52.231128973251124]
  We compare various strategies for predicting survival at the WSI and patient level. The former treats each WSI as an independent sample, mimicking the strategy adopted in other works. The latter comprises methods to either aggregate the predictions of the several WSIs or automatically identify the most relevant slide.
  arXiv  Detail & Related papers  (2025-03-29T11:14:02Z)
• Retinal Fundus Multi-Disease Image Classification using Hybrid CNN-Transformer-Ensemble Architectures [0.3277163122167434]
  Our research is motivated by the urgent global issue of a large population affected by retinal diseases. Our primary objective is to develop a comprehensive diagnostic system capable of accurately predicting retinal diseases.
  arXiv  Detail & Related papers  (2025-03-27T12:55:07Z)
• Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images [7.048241543461529]
  We propose a novel framework called Multi-Resolution Prompt-guided Hybrid Embedding (MR-PHE) to address these challenges in zero-shot histopathology image classification. We introduce a hybrid embedding strategy that integrates global image embeddings with weighted patch embeddings. A similarity-based patch weighting mechanism assigns attention-like weights to patches based on their relevance to class embeddings.
  arXiv  Detail & Related papers  (2025-03-13T12:18:37Z)
• OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology [7.743511021846898]
  Large language models (LLMs) have shown significant promise across various medical applications. We introduce the OphthBench, a benchmark designed to assess LLM performance within the context of Chinese ophthalmic practices. This framework allows for a thorough assessment of LLMs' capabilities and provides insights into their practical application in Chinese ophthalmology.
  arXiv  Detail & Related papers  (2025-02-03T11:04:51Z)
• Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering [70.44269982045415]
  Retrieval-augmented generation (RAG) has emerged as a promising approach to enhance the performance of large language models (LLMs) We introduce Medical Retrieval-Augmented Generation Benchmark (MedRGB) that provides various supplementary elements to four medical QA datasets. Our experimental results reveals current models' limited ability to handle noise and misinformation in the retrieved documents.
  arXiv  Detail & Related papers  (2024-11-14T06:19:18Z)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI [67.09501109871351]
  Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals. GMAI-MMBench is the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format.
  arXiv  Detail & Related papers  (2024-08-06T17:59:21Z)
• Optimizing Skin Lesion Classification via Multimodal Data and Auxiliary Task Integration [54.76511683427566]
  This research introduces a novel multimodal method for classifying skin lesions, integrating smartphone-captured images with essential clinical and demographic information. A distinctive aspect of this method is the integration of an auxiliary task focused on super-resolution image prediction. The experimental evaluations have been conducted using the PAD-UFES20 dataset, applying various deep-learning architectures.
  arXiv  Detail & Related papers  (2024-02-16T05:16:20Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM [48.16696073640864]
  We introduce OmniMedVQA, a novel comprehensive medical Visual Question Answering (VQA) benchmark. All images in this benchmark are sourced from authentic medical scenarios. We have found that existing LVLMs struggle to address these medical VQA problems effectively.
  arXiv  Detail & Related papers  (2024-02-14T13:51:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.