Decoupling Continual Semantic Segmentation

• URL: http://arxiv.org/abs/2508.05065v1
• Date: Thu, 07 Aug 2025 06:34:34 GMT
• Title: Decoupling Continual Semantic Segmentation
• Authors: Yifu Guo, Yuquan Lu, Wentao Zhang, Zishan Xu, Dexia Chen, Siyu Zhang, Yizhe Zhang, Ruixuan Wang,
• Abstract summary: Continual Semantics (CSS) requires learning new classes without forgetting previously acquired knowledge.<n>We introduce DecoupleCSS, a novel two-stage framework for CSS.<n>By decoupling class-aware detection from class-agnostic segmentation, DecoupleCSS enables more effective continual learning.
• Score: 25.237836663570913
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual Semantic Segmentation (CSS) requires learning new classes without forgetting previously acquired knowledge, addressing the fundamental challenge of catastrophic forgetting in dense prediction tasks. However, existing CSS methods typically employ single-stage encoder-decoder architectures where segmentation masks and class labels are tightly coupled, leading to interference between old and new class learning and suboptimal retention-plasticity balance. We introduce DecoupleCSS, a novel two-stage framework for CSS. By decoupling class-aware detection from class-agnostic segmentation, DecoupleCSS enables more effective continual learning, preserving past knowledge while learning new classes. The first stage leverages pre-trained text and image encoders, adapted using LoRA, to encode class-specific information and generate location-aware prompts. In the second stage, the Segment Anything Model (SAM) is employed to produce precise segmentation masks, ensuring that segmentation knowledge is shared across both new and previous classes. This approach improves the balance between retention and adaptability in CSS, achieving state-of-the-art performance across a variety of challenging tasks. Our code is publicly available at: https://github.com/euyis1019/Decoupling-Continual-Semantic-Segmentation.

Related papers

• Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning [64.1745161657794]
  Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains.<n>Recent advances in pre-trained models provide a solid foundation for DIL.<n>However, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge.<n>We propose DUal ConsolidaTion (Duct) to unify and consolidate historical knowledge.
  arXiv  Detail & Related papers  (2024-10-01T17:58:06Z)
• Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling [21.114359437665364]
  Continual semantic segmentation (CSS) based on incremental learning (IL) is a great endeavour in developing human-like segmentation models. Current CSS approaches encounter challenges in the trade-off between preserving old knowledge and learning new ones. We present Learning at a Glance (LAG), an efficient, robust, human-like and interpretable approach for CSS.
  arXiv  Detail & Related papers  (2024-07-22T07:17:52Z)
• BACS: Background Aware Continual Semantic Segmentation [15.821935479975343]
  In autonomous driving, there's a need to incorporate new classes as the operating environment of the deployed agent becomes more complex. For enhanced annotation efficiency, ideally, only pixels belonging to new classes would be annotated. This paper proposes a Backward Background Shift Detector (BACS) to detect previously observed classes.
  arXiv  Detail & Related papers  (2024-04-19T19:25:26Z)
• Enhancing Visual Continual Learning with Language-Guided Supervision [76.38481740848434]
  Continual learning aims to empower models to learn new tasks without forgetting previously acquired knowledge. We argue that the scarce semantic information conveyed by the one-hot labels hampers the effective knowledge transfer across tasks. Specifically, we use PLMs to generate semantic targets for each class, which are frozen and serve as supervision signals.
  arXiv  Detail & Related papers  (2024-03-24T12:41:58Z)
• Boosting Semantic Segmentation from the Perspective of Explicit Class Embeddings [19.997929884477628]
  We explore the mechanism of class embeddings and have an insight that more explicit and meaningful class embeddings can be generated based on class masks purposely. We propose ECENet, a new segmentation paradigm, in which class embeddings are obtained and enhanced explicitly during interacting with multi-stage image features. Our ECENet outperforms its counterparts on the ADE20K dataset with much less computational cost and achieves new state-of-the-art results on PASCAL-Context dataset.
  arXiv  Detail & Related papers  (2023-08-24T16:16:10Z)
• Visual and Textual Prior Guided Mask Assemble for Few-Shot Segmentation and Beyond [0.0]
  We propose a visual and textual Prior Guided Mask Assemble Network (PGMA-Net) It employs a class-agnostic mask assembly process to alleviate the bias, and formulates diverse tasks into a unified manner by assembling the prior through affinity. It achieves new state-of-the-art results in the FSS task, with mIoU of $77.6$ on $textPASCAL-5i$ and $59.4$ on $textCOCO-20i$ in 1-shot scenario.
  arXiv  Detail & Related papers  (2023-08-15T02:46:49Z)
• DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation [73.54038780856554]
  Class Incremental Semantic (CISS) extends the traditional segmentation task by incrementally learning newly added classes. Previous work has introduced generative replay, which involves replaying old class samples generated from a pre-trained GAN. We propose DiffusePast, a novel framework featuring a diffusion-based generative replay module that generates semantically accurate images with more reliable masks guided by different instructions.
  arXiv  Detail & Related papers  (2023-08-02T13:13:18Z)
• AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learning [53.32576252950481]
  Continual learning aims to enable a model to incrementally learn knowledge from sequentially arrived data. In this paper, we propose a non-incremental learner, named AttriCLIP, to incrementally extract knowledge of new classes or tasks.
  arXiv  Detail & Related papers  (2023-05-19T07:39:17Z)
• Side Adapter Network for Open-Vocabulary Semantic Segmentation [69.18441687386733]
  This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN) A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias. Our approach significantly outperforms other counterparts, with up to 18 times fewer trainable parameters and 19 times faster inference speed.
  arXiv  Detail & Related papers  (2023-02-23T18:58:28Z)
• Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation [84.1985497426083]
  convolutional neural networks are ill-equipped for incremental learning. New classes are available but the initial training data is not retained. We try to address this issue by "inverting" the trained segmentation network to synthesize input images starting from random noise.
  arXiv  Detail & Related papers  (2021-04-02T03:47:16Z)
• PLOP: Learning without Forgetting for Continual Semantic Segmentation [44.49799311137856]
  Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level. We also design an entropy-based pseudo-labelling of the background w.r.t. classes predicted by the old model to deal with background shift and avoid catastrophic forgetting of the old classes.
  arXiv  Detail & Related papers  (2020-11-23T13:35:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.