Towards a text-based quantitative and explainable histopathology image analysis

• URL: http://arxiv.org/abs/2407.07360v1
• Date: Wed, 10 Jul 2024 04:33:43 GMT
• Title: Towards a text-based quantitative and explainable histopathology image analysis
• Authors: Anh Tien Nguyen, Trinh Thi Le Vuong, Jin Tae Kwak,
• Abstract summary: We propose a Text-based Quantitative and Explainable histopathology image analysis, which we call TQx. The retrieved words are then used to quantify the histopathology images and generate understandable feature embeddings. The results demonstrate that TQx is able to quantify and analyze histopathology images comparable to the prevalent visual models in computational pathology.
• Score: 4.064178811354613
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, vision-language pre-trained models have emerged in computational pathology. Previous works generally focused on the alignment of image-text pairs via the contrastive pre-training paradigm. Such pre-trained models have been applied to pathology image classification in zero-shot learning or transfer learning fashion. Herein, we hypothesize that the pre-trained vision-language models can be utilized for quantitative histopathology image analysis through a simple image-to-text retrieval. To this end, we propose a Text-based Quantitative and Explainable histopathology image analysis, which we call TQx. Given a set of histopathology images, we adopt a pre-trained vision-language model to retrieve a word-of-interest pool. The retrieved words are then used to quantify the histopathology images and generate understandable feature embeddings due to the direct mapping to the text description. To evaluate the proposed method, the text-based embeddings of four histopathology image datasets are utilized to perform clustering and classification tasks. The results demonstrate that TQx is able to quantify and analyze histopathology images that are comparable to the prevalent visual models in computational pathology.

Related papers

• CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment [40.811510317145675]
  CPLIP is a new unsupervised technique to enhance the alignment of images and text in histopathology. evaluated across multiple histopathology tasks, CPLIP shows notable improvements in zero-shot learning scenarios. To encourage further research and replication, the code for CPLIP is available on GitHub.
  arXiv  Detail & Related papers  (2024-06-07T18:39:58Z)
• Renal digital pathology visual knowledge search platform based on language large model and book knowledge [0.1398098625978622]
  We conducted image segmentation and paired corresponding text descriptions based on 60 books for renal pathology. We evaluated the semantic feature capabilities of 4 large models, including GPT2, gemma, LLma and Qwen. We built a semantic retrieval system to retrieve pathological images based on text descriptions, and named RppD.
  arXiv  Detail & Related papers  (2024-05-27T01:03:12Z)
• Knowledge-enhanced Visual-Language Pretraining for Computational Pathology [68.6831438330526]
  We consider the problem of visual representation learning for computational pathology, by exploiting large-scale image-text pairs gathered from public resources. We curate a pathology knowledge tree that consists of 50,470 informative attributes for 4,718 diseases requiring pathology diagnosis from 32 human tissues.
  arXiv  Detail & Related papers  (2024-04-15T17:11:25Z)
• Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning [64.1316997189396]
  We present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets.
  arXiv  Detail & Related papers  (2024-03-21T17:58:56Z)
• Text-guided Foundation Model Adaptation for Pathological Image Classification [40.45252665455015]
  We propose to connect image and text Embeddings (CITE) to enhance pathological image classification. CITE injects text insights gained from language models pre-trained with a broad range of biomedical texts, leading to adapt foundation models towards pathological image understanding.
  arXiv  Detail & Related papers  (2023-07-27T14:44:56Z)
• Towards a Visual-Language Foundation Model for Computational Pathology [5.72536252929528]
  We introduce CONtrastive learning from Captions for Histopathology (CONCH) CONCH is a visual-language foundation model developed using diverse sources of histopathology images, biomedical text, and task-agnostic pretraining. It is evaluated on a suite of 13 diverse benchmarks, achieving state-of-the-art performance on histology image classification, segmentation, captioning, text-to-image and image-to-text retrieval.
  arXiv  Detail & Related papers  (2023-07-24T16:13:43Z)
• Seeing in Words: Learning to Classify through Language Bottlenecks [59.97827889540685]
  Humans can explain their predictions using succinct and intuitive descriptions. We show that a vision model whose feature representations are text can effectively classify ImageNet images.
  arXiv  Detail & Related papers  (2023-06-29T00:24:42Z)
• Semantic Search for Large Scale Clinical Ontologies [63.71950996116403]
  We present a deep learning approach to build a search system for large clinical vocabularies. We propose a Triplet-BERT model and a method that generates training data based on semantic training data. The model is evaluated using five real benchmark data sets and the results show that our approach achieves high results on both free text to concept and concept to searching concept vocabularies.
  arXiv  Detail & Related papers  (2022-01-01T05:15:42Z)
• HistoCartography: A Toolkit for Graph Analytics in Digital Pathology [0.6299766708197883]
  HistoCartography is a standardized python API with necessary preprocessing, machine learning and explainability tools to facilitate graph-analytics in computational pathology. We have benchmarked the computational time and performance on multiple datasets across different imaging types and histopathology tasks.
  arXiv  Detail & Related papers  (2021-07-21T13:34:14Z)
• Semantic segmentation of multispectral photoacoustic images using deep learning [53.65837038435433]
  Photoacoustic imaging has the potential to revolutionise healthcare. Clinical translation of the technology requires conversion of the high-dimensional acquired data into clinically relevant and interpretable information. We present a deep learning-based approach to semantic segmentation of multispectral photoacoustic images.
  arXiv  Detail & Related papers  (2021-05-20T09:33:55Z)
• Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
  We propose a deep co-attention network for multi-view subspace learning. It aims to extract both the common information and the complementary information in an adversarial setting. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
  arXiv  Detail & Related papers  (2021-02-15T18:46:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.