ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Prompting

• URL: http://arxiv.org/abs/2312.04964v2
• Date: Fri, 22 Mar 2024 12:34:13 GMT
• Title: ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Prompting
• Authors: Yankai Jiang, Zhongzhen Huang, Rongzhao Zhang, Xiaofan Zhang, Shaoting Zhang,
• Abstract summary: We propose a new zero-shot pan-tumor segmentation framework (ZePT) based on query-disentangling and self-prompting to segment unseen tumor categories. ZePT disentangles the object queries into two subsets and trains them in two stages. Experiments on various tumor segmentation tasks demonstrate the performance superiority of ZePT.
• Score: 11.087654014615955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The long-tailed distribution problem in medical image analysis reflects a high prevalence of common conditions and a low prevalence of rare ones, which poses a significant challenge in developing a unified model capable of identifying rare or novel tumor categories not encountered during training. In this paper, we propose a new zero-shot pan-tumor segmentation framework (ZePT) based on query-disentangling and self-prompting to segment unseen tumor categories beyond the training set. ZePT disentangles the object queries into two subsets and trains them in two stages. Initially, it learns a set of fundamental queries for organ segmentation through an object-aware feature grouping strategy, which gathers organ-level visual features. Subsequently, it refines the other set of advanced queries that focus on the auto-generated visual prompts for unseen tumor segmentation. Moreover, we introduce query-knowledge alignment at the feature level to enhance each query's discriminative representation and generalizability. Extensive experiments on various tumor segmentation tasks demonstrate the performance superiority of ZePT, which surpasses the previous counterparts and evidence the promising ability for zero-shot tumor segmentation in real-world settings.

Related papers

• MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentation [2.2585213273821716]
  We introduce MedCLIP-SAMv2, a novel framework that integrates the CLIP and SAM models to perform segmentation on clinical scans. Our approach includes fine-tuning the BiomedCLIP model with a new Decoupled Hard Negative Noise Contrastive Estimation (DHN-NCE) loss. We also investigate using zero-shot segmentation labels within a weakly supervised paradigm to enhance segmentation quality further.
  arXiv  Detail & Related papers  (2024-09-28T23:10:37Z)
• A Weakly Supervised and Globally Explainable Learning Framework for Brain Tumor Segmentation [8.253446049933483]
  Machine-based brain tumor segmentation can help doctors make better diagnoses. The complex structure of brain tumors and expensive pixel-level annotations present challenges for automatic tumor segmentation. We propose a counterfactual generation framework that achieves exceptional brain tumor segmentation performance without the need for pixel-level annotations.
  arXiv  Detail & Related papers  (2024-08-02T11:18:32Z)
• CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation [11.087654014615955]
  We introduce CAT, an innovative model that Coordinates Anatomical prompts derived from 3D cropped images with Textual prompts enriched by medical domain knowledge. Trained on a consortium of 10 public CT datasets, CAT demonstrates superior performance in multiple segmentation tasks. This approach confirms that coordinating multimodal prompts is a promising avenue for addressing complex scenarios in the medical domain.
  arXiv  Detail & Related papers  (2024-06-11T09:22:39Z)
• Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation [39.81051783009144]
  Segment Any Cell (SAC) is an innovative framework that enhances SAM for nuclei segmentation. SAC integrates a Low-Rank Adaptation (LoRA) within the attention layer of the Transformer to improve the fine-tuning process. Our contributions include a novel prompt generation strategy, automated adaptability for diverse segmentation tasks, and a versatile framework for semantic segmentation challenges.
  arXiv  Detail & Related papers  (2024-01-24T04:23:17Z)
• Multi-task Explainable Skin Lesion Classification [54.76511683427566]
  We propose a few-shot-based approach for skin lesions that generalizes well with few labelled data. The proposed approach comprises a fusion of a segmentation network that acts as an attention module and classification network.
  arXiv  Detail & Related papers  (2023-10-11T05:49:47Z)
• Prototype-Driven and Multi-Expert Integrated Multi-Modal MR Brain Tumor Image Segmentation [23.127213250062425]
  The impact of information aliasing caused by the mutual inclusion of tumor sub-regions is often ignored. Existing methods usually do not take tailored efforts to highlight the single tumor sub-region features. We propose a multi-modal MR brain tumor segmentation method with tumor prototype-driven and multi-expert integration.
  arXiv  Detail & Related papers  (2023-07-22T22:41:11Z)
• Multi-class versus One-class classifier in spontaneous speech analysis oriented to Alzheimer Disease diagnosis [58.720142291102135]
  The aim of our project is to contribute to earlier diagnosis of AD and better estimates of its severity by using automatic analysis performed through new biomarkers extracted from speech signal. The use of information about outlier and Fractal Dimension features improves the system performance.
  arXiv  Detail & Related papers  (2022-03-21T09:57:20Z)
• Factored Attention and Embedding for Unstructured-view Topic-related Ultrasound Report Generation [70.7778938191405]
  We propose a novel factored attention and embedding model (termed FAE-Gen) for the unstructured-view topic-related ultrasound report generation. The proposed FAE-Gen mainly consists of two modules, i.e., view-guided factored attention and topic-oriented factored embedding, which capture the homogeneous and heterogeneous morphological characteristic across different views.
  arXiv  Detail & Related papers  (2022-03-12T15:24:03Z)
• Generalized Organ Segmentation by Imitating One-shot Reasoning using Anatomical Correlation [55.1248480381153]
  We propose OrganNet which learns a generalized organ concept from a set of annotated organ classes and then transfer this concept to unseen classes. We show that OrganNet can effectively resist the wide variations in organ morphology and produce state-of-the-art results in one-shot segmentation task.
  arXiv  Detail & Related papers  (2021-03-30T13:41:12Z)
• Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding [60.89561661441736]
  We propose a novel method for few-shot medical image segmentation. We construct our few-shot image segmentor using a deep convolutional network trained episodically. We enhance discriminability of deep embedding to encourage clustering of the feature domains of the same class.
  arXiv  Detail & Related papers  (2020-12-10T04:01:07Z)
• Weakly supervised multiple instance learning histopathological tumor segmentation [51.085268272912415]
  We propose a weakly supervised framework for whole slide imaging segmentation. We exploit a multiple instance learning scheme for training models. The proposed framework has been evaluated on multi-locations and multi-centric public data from The Cancer Genome Atlas and the PatchCamelyon dataset.
  arXiv  Detail & Related papers  (2020-04-10T13:12:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.