SATS: Self-Attention Transfer for Continual Semantic Segmentation

• URL: http://arxiv.org/abs/2203.07667v1
• Date: Tue, 15 Mar 2022 06:09:28 GMT
• Title: SATS: Self-Attention Transfer for Continual Semantic Segmentation
• Authors: Yiqiao Qiu, Yixing Shen, Zhuohao Sun, Yanchong Zheng, Xiaobin Chang, Weishi Zheng, and Ruixuan Wang
• Abstract summary: continual semantic segmentation suffers from the same catastrophic forgetting issue as in continual classification learning. This study proposes to transfer a new type of information relevant to knowledge, i.e. the relationships between elements within each image. The relationship information can be effectively obtained from the self-attention maps in a Transformer-style segmentation model.
• Score: 50.51525791240729
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continually learning to segment more and more types of image regions is a desired capability for many intelligent systems. However, such continual semantic segmentation suffers from the same catastrophic forgetting issue as in continual classification learning. While multiple knowledge distillation strategies originally for continual classification have been well adapted to continual semantic segmentation, they only consider transferring old knowledge based on the outputs from one or more layers of deep fully convolutional networks. Different from existing solutions, this study proposes to transfer a new type of information relevant to knowledge, i.e. the relationships between elements (Eg. pixels or small local regions) within each image which can capture both within-class and between-class knowledge. The relationship information can be effectively obtained from the self-attention maps in a Transformer-style segmentation model. Considering that pixels belonging to the same class in each image often share similar visual properties, a class-specific region pooling is applied to provide more efficient relationship information for knowledge transfer. Extensive evaluations on multiple public benchmarks support that the proposed self-attention transfer method can further effectively alleviate the catastrophic forgetting issue, and its flexible combination with one or more widely adopted strategies significantly outperforms state-of-the-art solu

Related papers

• A Classifier-Free Incremental Learning Framework for Scalable Medical Image Segmentation [6.591403935303867]
  We introduce a novel segmentation paradigm enabling the segmentation of a variable number of classes within a single classifier-free network. This network is trained using contrastive learning and produces discriminative feature representations that facilitate straightforward interpretation. We demonstrate the flexibility of our method in handling varying class numbers within a unified network and its capacity for incremental learning.
  arXiv  Detail & Related papers  (2024-05-25T19:05:07Z)
• AF$_2$: Adaptive Focus Framework for Aerial Imagery Segmentation [86.44683367028914]
  Aerial imagery segmentation has some unique challenges, the most critical one among which lies in foreground-background imbalance. We propose Adaptive Focus Framework (AF$), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multi-scale representations. AF$ has significantly improved the accuracy on three widely used aerial benchmarks, as fast as the mainstream method.
  arXiv  Detail & Related papers  (2022-02-18T10:14:45Z)
• Long-tail Recognition via Compositional Knowledge Transfer [60.03764547406601]
  We introduce a novel strategy for long-tail recognition that addresses the tail classes' few-shot problem. Our objective is to transfer knowledge acquired from information-rich common classes to semantically similar, and yet data-hungry, rare classes. Experiments show that our approach can achieve significant performance boosts on rare classes while maintaining robust common class performance.
  arXiv  Detail & Related papers  (2021-12-13T15:48:59Z)
• A Contrastive Distillation Approach for Incremental Semantic Segmentation in Aerial Images [15.75291664088815]
  A major issue concerning current deep neural architectures is known as catastrophic forgetting. We propose a contrastive regularization, where any given input is compared with its augmented version. We show the effectiveness of our solution on the Potsdam dataset, outperforming the incremental baseline in every test.
  arXiv  Detail & Related papers  (2021-12-07T16:44:45Z)
• Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation [53.49821324597837]
  Weakly supervised semantic segmentation is a challenging problem that has been deeply studied in recent years. We present a Context Decoupling Augmentation ( CDA) method to change the inherent context in which the objects appear. To validate the effectiveness of the proposed method, extensive experiments on PASCAL VOC 2012 dataset with several alternative network architectures demonstrate that CDA can boost various popular WSSS methods to the new state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2021-03-02T15:05:09Z)
• Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation [128.03739769844736]
  Two neural co-attentions are incorporated into the classifier to capture cross-image semantic similarities and differences. In addition to boosting object pattern learning, the co-attention can leverage context from other related images to improve localization map inference. Our algorithm sets new state-of-the-arts on all these settings, demonstrating well its efficacy and generalizability.
  arXiv  Detail & Related papers  (2020-07-03T21:53:46Z)
• Weakly-Supervised Semantic Segmentation by Iterative Affinity Learning [86.45526827323954]
  Weakly-supervised semantic segmentation is a challenging task as no pixel-wise label information is provided for training. We propose an iterative algorithm to learn such pairwise relations. We show that the proposed algorithm performs favorably against the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-02-19T10:32:03Z)
• Continual Local Replacement for Few-shot Learning [13.956960291580938]
  The goal of few-shot learning is to learn a model that can recognize novel classes based on one or few training data. It is challenging mainly due to two aspects: (1) it lacks good feature representation of novel classes; (2) a few of labeled data could not accurately represent the true data distribution. A novel continual local replacement strategy is proposed to address the data deficiency problem.
  arXiv  Detail & Related papers  (2020-01-23T04:26:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.