Renal digital pathology visual knowledge search platform based on language large model and book knowledge

• URL: http://arxiv.org/abs/2406.18556v1
• Date: Mon, 27 May 2024 01:03:12 GMT
• Title: Renal digital pathology visual knowledge search platform based on language large model and book knowledge
• Authors: Xiaomin Lv, Chong Lai, Liya Ding, Maode Lai, Qingrong Sun,
• Abstract summary: We conducted image segmentation and paired corresponding text descriptions based on 60 books for renal pathology. We evaluated the semantic feature capabilities of 4 large models, including GPT2, gemma, LLma and Qwen. We built a semantic retrieval system to retrieve pathological images based on text descriptions, and named RppD.
• Score: 0.1398098625978622
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large models have become mainstream, yet their applications in digital pathology still require exploration. Meanwhile renal pathology images play an important role in the diagnosis of renal diseases. We conducted image segmentation and paired corresponding text descriptions based on 60 books for renal pathology, clustering analysis for all image and text description features based on large models, ultimately building a retrieval system based on the semantic features of large models. Based above analysis, we established a knowledge base of 10,317 renal pathology images and paired corresponding text descriptions, and then we evaluated the semantic feature capabilities of 4 large models, including GPT2, gemma, LLma and Qwen, and the image-based feature capabilities of dinov2 large model. Furthermore, we built a semantic retrieval system to retrieve pathological images based on text descriptions, and named RppD (aidp.zjsru.edu.cn).

Related papers

• Towards a text-based quantitative and explainable histopathology image analysis [4.064178811354613]
  We propose a Text-based Quantitative and Explainable histopathology image analysis, which we call TQx. The retrieved words are then used to quantify the histopathology images and generate understandable feature embeddings. The results demonstrate that TQx is able to quantify and analyze histopathology images comparable to the prevalent visual models in computational pathology.
  arXiv  Detail & Related papers  (2024-07-10T04:33:43Z)
• Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
  We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
  arXiv  Detail & Related papers  (2024-04-27T05:03:42Z)
• Knowledge-enhanced Visual-Language Pretraining for Computational Pathology [68.6831438330526]
  We consider the problem of visual representation learning for computational pathology, by exploiting large-scale image-text pairs gathered from public resources. We curate a pathology knowledge tree that consists of 50,470 informative attributes for 4,718 diseases requiring pathology diagnosis from 32 human tissues. All codes, models and the pathology knowledge tree will be released to the research community.
  arXiv  Detail & Related papers  (2024-04-15T17:11:25Z)
• Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning [64.1316997189396]
  We present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets.
  arXiv  Detail & Related papers  (2024-03-21T17:58:56Z)
• In-context learning enables multimodal large language models to classify cancer pathology images [0.7085801706650957]
  In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples.
  arXiv  Detail & Related papers  (2024-03-12T08:34:34Z)
• Leveraging Foundation Models for Content-Based Medical Image Retrieval in Radiology [0.14631663747888957]
  Content-based image retrieval has the potential to significantly improve diagnostic aid and medical research in radiology. Current CBIR systems face limitations due to their specialization to certain pathologies, limiting their utility. We propose using vision foundation models as powerful and versatile off-the-shelf feature extractors for content-based medical image retrieval.
  arXiv  Detail & Related papers  (2024-03-11T10:06:45Z)
• Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models [49.95603725998561]
  We propose a new paradigm to build robust and interpretable medical image classifiers with natural language concepts. Specifically, we first query clinical concepts from GPT-4, then transform latent image features into explicit concepts with a vision-language model.
  arXiv  Detail & Related papers  (2023-10-04T21:57:09Z)
• Towards a Visual-Language Foundation Model for Computational Pathology [5.72536252929528]
  We introduce CONtrastive learning from Captions for Histopathology (CONCH) CONCH is a visual-language foundation model developed using diverse sources of histopathology images, biomedical text, and task-agnostic pretraining. It is evaluated on a suite of 13 diverse benchmarks, achieving state-of-the-art performance on histology image classification, segmentation, captioning, text-to-image and image-to-text retrieval.
  arXiv  Detail & Related papers  (2023-07-24T16:13:43Z)
• Region-based Contrastive Pretraining for Medical Image Retrieval with Anatomic Query [56.54255735943497]
  Region-based contrastive pretraining for Medical Image Retrieval (RegionMIR) We introduce a novel Region-based contrastive pretraining for Medical Image Retrieval (RegionMIR)
  arXiv  Detail & Related papers  (2023-05-09T16:46:33Z)
• Medical Image Captioning via Generative Pretrained Transformers [57.308920993032274]
  We combine two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The proposed model is tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO.
  arXiv  Detail & Related papers  (2022-09-28T10:27:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.