Improving mitosis detection on histopathology images using large vision-language models

• URL: http://arxiv.org/abs/2310.07176v1
• Date: Wed, 11 Oct 2023 04:00:17 GMT
• Title: Improving mitosis detection on histopathology images using large vision-language models
• Abstract summary: In certain types of cancerous tissue, mitotic count has been shown to be associated with tumor proliferation, poor prognosis, and therapeutic resistance. Due to the high inter-rater variability of mitotic counting by pathologists, convolutional neural networks (CNNs) have been employed to reduce the subjectivity of mitosis detection in hematoxylin and eosin (H&E)-stained whole slide images. In this work, we demonstrate that pre-trained large-scale vision-language models that leverage both visual features and natural language improve mitosis detection accuracy.
• Score: 0.6144680854063939
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In certain types of cancerous tissue, mitotic count has been shown to be associated with tumor proliferation, poor prognosis, and therapeutic resistance. Due to the high inter-rater variability of mitotic counting by pathologists, convolutional neural networks (CNNs) have been employed to reduce the subjectivity of mitosis detection in hematoxylin and eosin (H&E)-stained whole slide images. However, most existing models have performance that lags behind expert panel review and only incorporate visual information. In this work, we demonstrate that pre-trained large-scale vision-language models that leverage both visual features and natural language improve mitosis detection accuracy. We formulate the mitosis detection task as an image captioning task and a visual question answering (VQA) task by including metadata such as tumor and scanner types as context. The effectiveness of our pipeline is demonstrated via comparison with various baseline models using 9,501 mitotic figures and 11,051 hard negatives (non-mitotic figures that are difficult to characterize) from the publicly available Mitosis Domain Generalization Challenge (MIDOG22) dataset.

Related papers

• BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases [59.19934491064188]
  Artificial intelligence (AI) has achieved remarkable success in medical imaging.<n>These models often perform inconsistently across real-world clinical settings.<n>We develop a benchmark of 85,355 CT scans that systematically evaluates 12 tumor-detection AI models.
  arXiv  Detail & Related papers  (2026-06-23T17:58:59Z)
• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• AGGRNet: Selective Feature Extraction and Aggregation for Enhanced Medical Image Classification [11.697160779548504]
  We propose AGGRNet framework to extract informative and non-informative features to understand fine-grained visual patterns.<n>Our model achieves state-of-the-art performance on various medical imaging datasets, with the best improvement up to 5% over SOTA models on the Kvasir dataset.
  arXiv  Detail & Related papers  (2025-11-15T23:01:09Z)
• Perceptual Evaluation of GANs and Diffusion Models for Generating X-rays [0.7578439720012189]
  Generative image models have achieved remarkable progress in both natural and medical imaging.<n>We evaluate the effectiveness of state-of-the-art generative models-Generative Adversarial Networks (GANs) and Diffusion Models (DMs)-for synthesizing chest X-rays conditioned on four abnormalities.
  arXiv  Detail & Related papers  (2025-08-10T00:32:18Z)
• RetinaLogos: Fine-Grained Synthesis of High-Resolution Retinal Images Through Captions [16.85664533914851]
  Existing methods for synthesising Colour Fundus Photographs largely rely on predefined disease labels.<n>We first introduce an innovative pipeline that creates a large-scale, captioned retinal dataset comprising 1.4 million entries.<n>We employ a novel three-step training framework, RetinaLogos, which enables fine-grained semantic control over retinal images.
  arXiv  Detail & Related papers  (2025-05-19T09:18:11Z)
• Bayesian Deep Learning Approaches for Uncertainty-Aware Retinal OCT Image Segmentation for Multiple Sclerosis [0.0]
  Optical Coherence Tomography ( OCT) provides valuable insights in ophthalmology, cardiology, and neurology.<n>One critical task for ophthalmologists using OCT is delineation of retinal layers within scans.<n>Previous efforts to automate delineation using deep learning face challenges in uptake from clinicians and statisticians.
  arXiv  Detail & Related papers  (2025-05-17T15:56:17Z)
• Prompting Medical Vision-Language Models to Mitigate Diagnosis Bias by Generating Realistic Dermoscopic Images [0.31077024712075796]
  Dermatology Diffusion Transformer (DermDiT) We propose a novel generative AI-based framework, which leverages text prompts generated via Vision Language Models and multimodal text-image learning to generate new dermoscopic images.
  arXiv  Detail & Related papers  (2025-04-02T15:44:12Z)
• Style transfer as data augmentation: evaluating unpaired image-to-image translation models in mammography [0.0]
  Deep learning models can learn to detect breast cancer from mammograms. However, challenges with overfitting and poor generalisability prevent their routine use in the clinic. Data augmentation techniques can be used to improve generalisability.
  arXiv  Detail & Related papers  (2025-02-04T16:52:45Z)
• Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis [55.959002385347645]
  Latent Drifting enables diffusion models to be conditioned for medical images fitted for the complex task of counterfactual image generation. We evaluate our method on three public longitudinal benchmark datasets of brain MRI and chest X-rays for counterfactual image generation.
  arXiv  Detail & Related papers  (2024-12-30T01:59:34Z)
• Unleashing the Potential of Synthetic Images: A Study on Histopathology Image Classification [0.12499537119440242]
  Histopathology image classification is crucial for the accurate identification and diagnosis of various diseases. We show that synthetic images can effectively augment existing datasets, ultimately improving the performance of the downstream histopathology image classification task.
  arXiv  Detail & Related papers  (2024-09-24T12:02:55Z)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images [68.42215385041114]
  This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-19T09:28:19Z)
• Could We Generate Cytology Images from Histopathology Images? An Empirical Study [1.791005104399795]
  In this study, we have explored traditional image-to-image transfer models like CycleGAN, and Neural Style Transfer. In this study, we have explored traditional image-to-image transfer models like CycleGAN, and Neural Style Transfer.
  arXiv  Detail & Related papers  (2024-03-16T10:43:12Z)
• Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generation [47.250147322130545]
  Image-to-text radiology report generation aims to automatically produce radiology reports that describe the findings in medical images. Most existing methods focus solely on the image data, disregarding the other patient information accessible to radiologists. We present a novel multi-modal deep neural network framework for generating chest X-rays reports by integrating structured patient data, such as vital signs and symptoms, alongside unstructured clinical notes.
  arXiv  Detail & Related papers  (2023-11-18T14:37:53Z)
• Semantic Map Guided Synthesis of Wireless Capsule Endoscopy Images using Diffusion Models [4.187344935012482]
  Wireless capsule endoscopy (WCE) is a non-invasive method for visualizing the gastrointestinal (GI) tract. We propose a novel approach leveraging generative models, specifically the diffusion model (DM), for generating diverse WCE images. Our model incorporates semantic map resulted from visualization scale (VS) engine, enhancing the controllability and diversity of generated images.
  arXiv  Detail & Related papers  (2023-11-10T06:16:44Z)
• Improving Deep Facial Phenotyping for Ultra-rare Disorder Verification Using Model Ensembles [52.77024349608834]
  We analyze the influence of replacing a DCNN with a state-of-the-art face recognition approach, iResNet with ArcFace. Our proposed ensemble model achieves state-of-the-art performance on both seen and unseen disorders.
  arXiv  Detail & Related papers  (2022-11-12T23:28:54Z)
• Data-Efficient Vision Transformers for Multi-Label Disease Classification on Chest Radiographs [55.78588835407174]
  Vision Transformers (ViTs) have not been applied to this task despite their high classification performance on generic images. ViTs do not rely on convolutions but on patch-based self-attention and in contrast to CNNs, no prior knowledge of local connectivity is present. Our results show that while the performance between ViTs and CNNs is on par with a small benefit for ViTs, DeiTs outperform the former if a reasonably large data set is available for training.
  arXiv  Detail & Related papers  (2022-08-17T09:07:45Z)
• Texture Characterization of Histopathologic Images Using Ecological Diversity Measures and Discrete Wavelet Transform [82.53597363161228]
  This paper proposes a method for characterizing texture across histopathologic images with a considerable success rate. It is possible to quantify the intrinsic properties of such images with promising accuracy on two HI datasets.
  arXiv  Detail & Related papers  (2022-02-27T02:19:09Z)
• Deep Learning models for benign and malign Ocular Tumor Growth Estimation [3.1558405181807574]
  Clinicians often face issues in selecting suitable image processing algorithm for medical imaging data. A strategy for the selection of a proper model is presented here.
  arXiv  Detail & Related papers  (2021-07-09T05:40:25Z)
• Image Translation for Medical Image Generation -- Ischemic Stroke Lesions [0.0]
  Synthetic databases with annotated pathologies could provide the required amounts of training data. We train different image-to-image translation models to synthesize magnetic resonance images of brain volumes with and without stroke lesions. We show that for a small database of only 10 or 50 clinical cases, synthetic data augmentation yields significant improvement.
  arXiv  Detail & Related papers  (2020-10-05T09:12:28Z)
• Towards Unsupervised Learning for Instrument Segmentation in Robotic Surgery with Cycle-Consistent Adversarial Networks [54.00217496410142]
  We propose an unpaired image-to-image translation where the goal is to learn the mapping between an input endoscopic image and a corresponding annotation. Our approach allows to train image segmentation models without the need to acquire expensive annotations. We test our proposed method on Endovis 2017 challenge dataset and show that it is competitive with supervised segmentation methods.
  arXiv  Detail & Related papers  (2020-07-09T01:39:39Z)
• Spectral-Spatial Recurrent-Convolutional Networks for In-Vivo Hyperspectral Tumor Type Classification [49.32653090178743]
  We demonstrate the feasibility of in-vivo tumor type classification using hyperspectral imaging and deep learning. Our best model achieves an AUC of 76.3%, significantly outperforming previous conventional and deep learning methods.
  arXiv  Detail & Related papers  (2020-07-02T12:00:53Z)
• An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization [45.00998416720726]
  We propose a framework to address the unique properties of medical images. This model first uses a low-capacity, yet memory-efficient, network on the whole image to identify the most informative regions. It then applies another higher-capacity network to collect details from chosen regions. Finally, it employs a fusion module that aggregates global and local information to make a final prediction.
  arXiv  Detail & Related papers  (2020-02-13T15:28:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.