Multiscale Progressive Text Prompt Network for Medical Image Segmentation

• URL: http://arxiv.org/abs/2307.00174v1
• Date: Fri, 30 Jun 2023 23:37:16 GMT
• Title: Multiscale Progressive Text Prompt Network for Medical Image Segmentation
• Authors: Xianjun Han, Qianqian Chen, Zhaoyang Xie, Xuejun Li, Hongyu Yang
• Abstract summary: We propose using progressive text prompts as prior knowledge to guide the segmentation process. Our model achieves high-quality results with low data annotation costs.
• Score: 10.121625177837931
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The accurate segmentation of medical images is a crucial step in obtaining reliable morphological statistics. However, training a deep neural network for this task requires a large amount of labeled data to ensure high-accuracy results. To address this issue, we propose using progressive text prompts as prior knowledge to guide the segmentation process. Our model consists of two stages. In the first stage, we perform contrastive learning on natural images to pretrain a powerful prior prompt encoder (PPE). This PPE leverages text prior prompts to generate multimodality features. In the second stage, medical image and text prior prompts are sent into the PPE inherited from the first stage to achieve the downstream medical image segmentation task. A multiscale feature fusion block (MSFF) combines the features from the PPE to produce multiscale multimodality features. These two progressive features not only bridge the semantic gap but also improve prediction accuracy. Finally, an UpAttention block refines the predicted results by merging the image and text features. This design provides a simple and accurate way to leverage multiscale progressive text prior prompts for medical image segmentation. Compared with using only images, our model achieves high-quality results with low data annotation costs. Moreover, our model not only has excellent reliability and validity on medical images but also performs well on natural images. The experimental results on different image datasets demonstrate that our model is effective and robust for image segmentation.

Related papers

• LoGra-Med: Long Context Multi-Graph Alignment for Medical Vision-Language Model [55.80651780294357]
  State-of-the-art medical multi-modal large language models (med-MLLM) leverage instruction-following data in pre-training. LoGra-Med is a new multi-graph alignment algorithm that enforces triplet correlations across image modalities, conversation-based descriptions, and extended captions. Our results show LoGra-Med matches LLAVA-Med performance on 600K image-text pairs for Medical VQA and significantly outperforms it when trained on 10% of the data.
  arXiv  Detail & Related papers  (2024-10-03T15:52:03Z)
• PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation [51.509573838103854]
  We propose a semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation. Our PMT generates high-fidelity pseudo labels by learning robust and diverse features in the training process. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches.
  arXiv  Detail & Related papers  (2024-09-08T15:02:25Z)
• XLIP: Cross-modal Attention Masked Modelling for Medical Language-Image Pre-Training [29.02600107837688]
  Vision-and-language pretraining uses contrastive learning on image-text pairs to achieve effective transfer across tasks. Current models struggle to accurately reconstruct key pathological features due to the scarcity of medical data. This paper proposes a XLIP (Masked modelling for medical Language-Image Pre-training) framework to enhance pathological learning and feature learning via unpaired data.
  arXiv  Detail & Related papers  (2024-07-28T17:38:21Z)
• MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training [103.72844619581811]
  We build performant Multimodal Large Language Models (MLLMs) In particular, we study the importance of various architecture components and data choices. We demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data.
  arXiv  Detail & Related papers  (2024-03-14T17:51:32Z)
• MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning [20.33625985769796]
  Existing contrastive language-image pre-training aims to learn a joint representation by matching abundant image-text pairs. We propose a Medical Language-Image Pre-training framework, which exploits the limited image-text medical data more efficiently. Our evaluation results show that MLIP outperforms previous work in zero/few-shot classification and few-shot segmentation tasks by a large margin.
  arXiv  Detail & Related papers  (2024-01-03T07:54:13Z)
• Self-Prompting Large Vision Models for Few-Shot Medical Image Segmentation [14.135249795318591]
  We propose a novel perspective on self-prompting in medical vision applications. We harness the embedding space of the Segment Anything Model to prompt itself through a simple yet effective linear pixel-wise classifier. We achieve competitive results on multiple datasets.
  arXiv  Detail & Related papers  (2023-08-15T08:20:07Z)
• Unified Medical Image-Text-Label Contrastive Learning With Continuous Prompt [3.218449686637963]
  We propose a unified Image-Text-Label contrastive learning framework based on continuous prompts. We demonstrate through sufficient experiments that the Unified Medical Contrastive Learning framework exhibits excellent performance on several downstream tasks.
  arXiv  Detail & Related papers  (2023-07-12T05:19:10Z)
• M$^{2}$SNet: Multi-scale in Multi-scale Subtraction Network for Medical Image Segmentation [73.10707675345253]
  We propose a general multi-scale in multi-scale subtraction network (M$2$SNet) to finish diverse segmentation from medical image. Our method performs favorably against most state-of-the-art methods under different evaluation metrics on eleven datasets of four different medical image segmentation tasks.
  arXiv  Detail & Related papers  (2023-03-20T06:26:49Z)
• Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts [63.84720380390935]
  There exist two typical types, textiti.e., the fusion-encoder type and the dual-encoder type, depending on whether a heavy fusion module is used. We propose an effective yet straightforward scheme named PTUnifier to unify the two types. We first unify the input format by introducing visual and textual prompts, which serve as a feature bank that stores the most representative images/texts.
  arXiv  Detail & Related papers  (2023-02-17T15:43:42Z)
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing [53.89917396428747]
  Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. We explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model.
  arXiv  Detail & Related papers  (2023-01-11T16:35:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.