Low-Rank Continual Pyramid Vision Transformer: Incrementally Segment Whole-Body Organs in CT with Light-Weighted Adaptation

• URL: http://arxiv.org/abs/2410.04689v1
• Date: Mon, 7 Oct 2024 02:00:13 GMT
• Title: Low-Rank Continual Pyramid Vision Transformer: Incrementally Segment Whole-Body Organs in CT with Light-Weighted Adaptation
• Authors: Vince Zhu, Zhanghexuan Ji, Dazhou Guo, Puyang Wang, Yingda Xia, Le Lu, Xianghua Ye, Wei Zhu, Dakai Jin,
• Abstract summary: We propose a new continual whole-body organ segmentation model with light-weighted low-rank adaptation (LoRA) We first train and freeze a pyramid vision transformer (PVT) base segmentation model on the initial task, then continually add light-weighted trainable LoRA parameters to the frozen model for each new learning task. Our proposed model continually segments new organs without catastrophic forgetting and meanwhile maintaining a low parameter increasing rate.
• Score: 10.746776960260297
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deep segmentation networks achieve high performance when trained on specific datasets. However, in clinical practice, it is often desirable that pretrained segmentation models can be dynamically extended to enable segmenting new organs without access to previous training datasets or without training from scratch. This would ensure a much more efficient model development and deployment paradigm accounting for the patient privacy and data storage issues. This clinically preferred process can be viewed as a continual semantic segmentation (CSS) problem. Previous CSS works would either experience catastrophic forgetting or lead to unaffordable memory costs as models expand. In this work, we propose a new continual whole-body organ segmentation model with light-weighted low-rank adaptation (LoRA). We first train and freeze a pyramid vision transformer (PVT) base segmentation model on the initial task, then continually add light-weighted trainable LoRA parameters to the frozen model for each new learning task. Through a holistically exploration of the architecture modification, we identify three most important layers (i.e., patch-embedding, multi-head attention and feed forward layers) that are critical in adapting to the new segmentation tasks, while retaining the majority of the pretrained parameters fixed. Our proposed model continually segments new organs without catastrophic forgetting and meanwhile maintaining a low parameter increasing rate. Continually trained and tested on four datasets covering different body parts of a total of 121 organs, results show that our model achieves high segmentation accuracy, closely reaching the PVT and nnUNet upper bounds, and significantly outperforms other regularization-based CSS methods. When comparing to the leading architecture-based CSS method, our model has a substantial lower parameter increasing rate while achieving comparable performance.

Related papers

• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• Incremental Learning for Heterogeneous Structure Segmentation in Brain Tumor MRI [11.314017805825685]
  We propose a divergence-aware dual-flow module with balanced rigidity and plasticity branches to decouple old and new tasks. We evaluate our framework on a brain tumor segmentation task with continually changing target domains.
  arXiv  Detail & Related papers  (2023-05-30T20:39:03Z)
• Prompt-Based Tuning of Transformer Models for Multi-Center Medical Image Segmentation of Head and Neck Cancer [0.0]
  This paper proposes a novel fine-tuning strategy for adapting a pretrained transformer-based segmentation model on data from a new medical center. Our strategy delivers great accuracy with minimum re-training on new-center data, significantly decreasing the computational and time costs.
  arXiv  Detail & Related papers  (2023-05-30T11:26:52Z)
• Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture [68.13678918660872]
  We design a more capable parameter-sharing architecture based on matrix product operator (MPO) MPO decomposition can reorganize and factorize the information of a parameter matrix into two parts. Our architecture shares the central tensor across all layers for reducing the model size.
  arXiv  Detail & Related papers  (2023-03-27T02:34:09Z)
• Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need [84.3507610522086]
  Class-incremental learning (CIL) aims to adapt to emerging new classes without forgetting old ones. Recent pre-training has achieved substantial progress, making vast pre-trained models (PTMs) accessible for CIL. We argue that the core factors in CIL are adaptivity for model updating and generalizability for knowledge transferring.
  arXiv  Detail & Related papers  (2023-03-13T17:59:02Z)
• Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans [31.388497540849297]
  We propose a new architectural CSS learning framework to learn a single deep segmentation model for segmenting a total of 143 whole-body organs. We trained and validated on 3D CT scans of 2500+ patients from four datasets, our single network can segment total 143 whole-body organs with very high accuracy.
  arXiv  Detail & Related papers  (2023-02-01T00:49:21Z)
• Prompt Tuning for Parameter-efficient Medical Image Segmentation [79.09285179181225]
  We propose and investigate several contributions to achieve a parameter-efficient but effective adaptation for semantic segmentation on two medical imaging datasets. We pre-train this architecture with a dedicated dense self-supervision scheme based on assignments to online generated prototypes. We demonstrate that the resulting neural network model is able to attenuate the gap between fully fine-tuned and parameter-efficiently adapted models.
  arXiv  Detail & Related papers  (2022-11-16T21:55:05Z)
• IterMiUnet: A lightweight architecture for automatic blood vessel segmentation [10.538564380139483]
  This paper proposes IterMiUnet, a new lightweight convolution-based segmentation model. It overcomes its heavily parametrized nature by incorporating the encoder-decoder structure of MiUnet model within it. The proposed model has a lot of potential to be utilized as a tool for the early diagnosis of many diseases.
  arXiv  Detail & Related papers  (2022-08-02T14:33:14Z)
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding [60.226644697970116]
  Domain classification is the fundamental task in natural language understanding (NLU) Most existing continual learning approaches suffer from low accuracy and performance fluctuation. We propose a hyper parameter-free continual learning model for text data that can stably produce high performance under various environments.
  arXiv  Detail & Related papers  (2022-01-05T02:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.