Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring

• URL: http://arxiv.org/abs/2405.08786v2
• Date: Wed, 10 Jul 2024 06:44:03 GMT
• Title: Incorporating Clinical Guidelines through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring
• Authors: Tiantian Zhang, Manxi Lin, Hongda Guo, Xiaofan Zhang, Ka Fung Peter Chiu, Aasa Feragen, Qi Dou,
• Abstract summary: Current deep learning-based PI-RADS scoring methods often lack the incorporation of common PI-RADS clinical guideline(PICG) utilized by radiologists. This paper introduces a novel approach that adapts a multi-modal large language model (MLLM) to incorporate PICG into PI-RADS scoring model without additional annotations and network parameters.
• Score: 14.804269296760712
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The Prostate Imaging Reporting and Data System (PI-RADS) is pivotal in the diagnosis of clinically significant prostate cancer through MRI imaging. Current deep learning-based PI-RADS scoring methods often lack the incorporation of common PI-RADS clinical guideline~(PICG) utilized by radiologists, potentially compromising scoring accuracy. This paper introduces a novel approach that adapts a multi-modal large language model (MLLM) to incorporate PICG into PI-RADS scoring model without additional annotations and network parameters. We present a designed two-stage fine-tuning process aiming at adapting a MLLM originally trained on natural images to the MRI images while effectively integrating the PICG. Specifically, in the first stage, we develop a domain adapter layer tailored for processing 3D MRI inputs and instruct the MLLM to differentiate MRI sequences. In the second stage, we translate PICG for guiding instructions from the model to generate PICG-guided image features. Through such a feature distillation step, we align the scoring network's features with the PICG-guided image features, which enables the model to effectively incorporate the PICG information. We develop our model on a public dataset and evaluate it on an in-house dataset. Experimental results demonstrate that our approach effectively improves the performance of current scoring networks. Code is available at: https://github.com/med-air/PICG2scoring

Related papers

• Mitigating Catastrophic Forgetting in the Incremental Learning of Medical Images [1.1510009152620668]
  This paper proposes an Incremental Learning (IL) approach to enhance the accuracy and efficiency of deep learning models in analyzing T2-weighted (T2w) MRI medical images prostate cancer detection. We used multiple health centers' artificial intelligence and radiology data, focused on different tasks that looked at prostate cancer detection using MRI (PI-CAI) We utilized Knowledge Distillation (KD) as it employs generated images from past tasks to guide the training of models for subsequent tasks.
  arXiv  Detail & Related papers  (2025-04-28T17:56:04Z)
• PathSegDiff: Pathology Segmentation using Diffusion model representations [63.20694440934692]
  We propose PathSegDiff, a novel approach for histopathology image segmentation that leverages Latent Diffusion Models (LDMs) as pre-trained featured extractors. Our method utilizes a pathology-specific LDM, guided by a self-supervised encoder, to extract rich semantic information from H&E stained histopathology images. Our experiments demonstrate significant improvements over traditional methods on the BCSS and GlaS datasets.
  arXiv  Detail & Related papers  (2025-04-09T14:58:21Z)
• A Mutual Inclusion Mechanism for Precise Boundary Segmentation in Medical Images [2.9137615132901704]
  We present a novel deep learning-based approach, MIPC-Net, for precise boundary segmentation in medical images. We introduce the MIPC module, which enhances the focus on channel information when extracting position features. We also propose the GL-MIPC-Residue, a global residual connection that enhances the integration of the encoder and decoder.
  arXiv  Detail & Related papers  (2024-04-12T02:14:35Z)
• NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation [55.51412454263856]
  This paper proposes to directly modulate the generation process of diffusion models using fMRI signals. By training with about 67,000 fMRI-image pairs from various individuals, our model enjoys superior fMRI-to-image decoding capacity.
  arXiv  Detail & Related papers  (2024-03-27T02:42:52Z)
• SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imaging [59.78761085714715]
  This study proposes a novel Siamese Dual-Resolution Transformer (SDR-Former) framework for liver lesion classification. The proposed framework has been validated through comprehensive experiments on two clinical datasets. To support the scientific community, we are releasing our extensive multi-phase MR dataset for liver lesion analysis to the public.
  arXiv  Detail & Related papers  (2024-02-27T06:32:56Z)
• Enhancing Prostate Cancer Diagnosis with Deep Learning: A Study using mpMRI Segmentation and Classification [0.0]
  Prostate cancer (PCa) is a severe disease among men globally. It is important to identify PCa early and make a precise diagnosis for effective treatment. Deep learning (DL) models can enhance existing clinical systems and improve patient care by locating regions of interest for physicians. This work uses well-known DL models for the classification and segmentation of mpMRI images to detect PCa.
  arXiv  Detail & Related papers  (2023-10-09T03:00:15Z)
• On the Localization of Ultrasound Image Slices within Point Distribution Models [84.27083443424408]
  Thyroid disorders are most commonly diagnosed using high-resolution Ultrasound (US) Longitudinal tracking is a pivotal diagnostic protocol for monitoring changes in pathological thyroid morphology. We present a framework for automated US image slice localization within a 3D shape representation.
  arXiv  Detail & Related papers  (2023-09-01T10:10:46Z)
• Explainable unsupervised multi-modal image registration using deep networks [2.197364252030876]
  MRI image registration aims to geometrically 'pair' diagnoses from different modalities, time points and slices. In this work, we show that our DL model becomes fully explainable, setting the framework to generalise our approach on further medical imaging data.
  arXiv  Detail & Related papers  (2023-08-03T19:13:48Z)
• Lesion-aware Dynamic Kernel for Polyp Segmentation [49.63274623103663]
  We propose a lesion-aware dynamic network (LDNet) for polyp segmentation. It is a traditional u-shape encoder-decoder structure incorporated with a dynamic kernel generation and updating scheme. This simple but effective scheme endows our model with powerful segmentation performance and generalization capability.
  arXiv  Detail & Related papers  (2023-01-12T09:53:57Z)
• Z-SSMNet: Zonal-aware Self-supervised Mesh Network for Prostate Cancer Detection and Diagnosis with Bi-parametric MRI [14.101371684361675]
  We propose a Zonal-aware Self-supervised Mesh Network (Z-SSMNet) Z-SSMNet adaptively integrates multi-dimensional (2D/2.5D/3D) convolutions to learn dense intra-slice information and sparse inter-slice information of the anisotropic bpMRI. A self-supervised learning (SSL) technique is proposed to pre-train our network using large-scale unlabeled data.
  arXiv  Detail & Related papers  (2022-12-12T10:08:46Z)
• Lung Cancer Lesion Detection in Histopathology Images Using Graph-Based Sparse PCA Network [93.22587316229954]
  We propose a graph-based sparse principal component analysis (GS-PCA) network, for automated detection of cancerous lesions on histological lung slides stained by hematoxylin and eosin (H&E) We evaluate the performance of the proposed algorithm on H&E slides obtained from an SVM K-rasG12D lung cancer mouse model using precision/recall rates, F-score, Tanimoto coefficient, and area under the curve (AUC) of the receiver operator characteristic (ROC)
  arXiv  Detail & Related papers  (2021-10-27T19:28:36Z)
• Optimising Knee Injury Detection with Spatial Attention and Validating Localisation Ability [0.5772546394254112]
  This work employs a pre-trained, multi-view Convolutional Neural Network (CNN) with a spatial attention block to optimise knee injury detection. An open-source Magnetic Resonance Imaging (MRI) data set with image-level labels was leveraged for this analysis.
  arXiv  Detail & Related papers  (2021-08-18T13:24:17Z)
• SAG-GAN: Semi-Supervised Attention-Guided GANs for Data Augmentation on Medical Images [47.35184075381965]
  We present a data augmentation method for generating synthetic medical images using cycle-consistency Generative Adversarial Networks (GANs) The proposed GANs-based model can generate a tumor image from a normal image, and in turn, it can also generate a normal image from a tumor image. We train the classification model using real images with classic data augmentation methods and classification models using synthetic images.
  arXiv  Detail & Related papers  (2020-11-15T14:01:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.