BACS: Background Aware Continual Semantic Segmentation

• URL: http://arxiv.org/abs/2404.13148v1
• Date: Fri, 19 Apr 2024 19:25:26 GMT
• Title: BACS: Background Aware Continual Semantic Segmentation
• Authors: Mostafa ElAraby, Ali Harakeh, Liam Paull,
• Abstract summary: In autonomous driving, there's a need to incorporate new classes as the operating environment of the deployed agent becomes more complex. For enhanced annotation efficiency, ideally, only pixels belonging to new classes would be annotated. This paper proposes a Backward Background Shift Detector (BACS) to detect previously observed classes.
• Score: 15.821935479975343
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic segmentation plays a crucial role in enabling comprehensive scene understanding for robotic systems. However, generating annotations is challenging, requiring labels for every pixel in an image. In scenarios like autonomous driving, there's a need to progressively incorporate new classes as the operating environment of the deployed agent becomes more complex. For enhanced annotation efficiency, ideally, only pixels belonging to new classes would be annotated. This approach is known as Continual Semantic Segmentation (CSS). Besides the common problem of classical catastrophic forgetting in the continual learning setting, CSS suffers from the inherent ambiguity of the background, a phenomenon we refer to as the "background shift'', since pixels labeled as background could correspond to future classes (forward background shift) or previous classes (backward background shift). As a result, continual learning approaches tend to fail. This paper proposes a Backward Background Shift Detector (BACS) to detect previously observed classes based on their distance in the latent space from the foreground centroids of previous steps. Moreover, we propose a modified version of the cross-entropy loss function, incorporating the BACS detector to down-weight background pixels associated with formerly observed classes. To combat catastrophic forgetting, we employ masked feature distillation alongside dark experience replay. Additionally, our approach includes a transformer decoder capable of adjusting to new classes without necessitating an additional classification head. We validate BACS's superior performance over existing state-of-the-art methods on standard CSS benchmarks.

Related papers

• Mitigating Background Shift in Class-Incremental Semantic Segmentation [18.604420743751643]
  Class-Incremental Semantic(CISS) aims to learn new classes without forgetting the old ones. We propose a background-class separation framework for CISS.
  arXiv  Detail & Related papers  (2024-07-16T15:44:37Z)
• DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation [73.54038780856554]
  Class Incremental Semantic (CISS) extends the traditional segmentation task by incrementally learning newly added classes. Previous work has introduced generative replay, which involves replaying old class samples generated from a pre-trained GAN. We propose DiffusePast, a novel framework featuring a diffusion-based generative replay module that generates semantically accurate images with more reliable masks guided by different instructions.
  arXiv  Detail & Related papers  (2023-08-02T13:13:18Z)
• TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale [59.01246141215051]
  We analyze the factor that leads to degradation from the perspective of language supervision. We propose a tunable-free pre-training strategy to retain the generalization ability of the text encoder. We produce a series of models, dubbed TVTSv2, with up to one billion parameters.
  arXiv  Detail & Related papers  (2023-05-23T15:44:56Z)
• RBC: Rectifying the Biased Context in Continual Semantic Segmentation [10.935529209436929]
  We propose a biased-context-rectified CSS framework with a context-rectified image-duplet learning scheme and a biased-context-insensitive consistency loss. Our approach outperforms state-of-the-art methods by a large margin in existing CSS scenarios.
  arXiv  Detail & Related papers  (2022-03-16T05:39:32Z)
• Weakly Supervised Semantic Segmentation using Out-of-Distribution Data [50.45689349004041]
  Weakly supervised semantic segmentation (WSSS) methods are often built on pixel-level localization maps. We propose a novel source of information to distinguish foreground from the background: Out-of-Distribution (OoD) data.
  arXiv  Detail & Related papers  (2022-03-08T05:33:35Z)
• Modeling the Background for Incremental and Weakly-Supervised Semantic Segmentation [39.025848280224785]
  We introduce a novel incremental class learning approach for semantic segmentation. Since each training step provides annotation only for a subset of all possible classes, pixels of the background class exhibit a semantic shift. We demonstrate the effectiveness of our approach with an extensive evaluation on the Pascal-VOC, ADE20K, and Cityscapes datasets.
  arXiv  Detail & Related papers  (2022-01-31T16:33:21Z)
• Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation [35.2461834832935]
  Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships. We also introduce a novel rehearsal method that is particularly suited for segmentation.
  arXiv  Detail & Related papers  (2021-06-29T11:57:21Z)
• Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation [84.1985497426083]
  convolutional neural networks are ill-equipped for incremental learning. New classes are available but the initial training data is not retained. We try to address this issue by "inverting" the trained segmentation network to synthesize input images starting from random noise.
  arXiv  Detail & Related papers  (2021-04-02T03:47:16Z)
• PLOP: Learning without Forgetting for Continual Semantic Segmentation [44.49799311137856]
  Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level. We also design an entropy-based pseudo-labelling of the background w.r.t. classes predicted by the old model to deal with background shift and avoid catastrophic forgetting of the old classes.
  arXiv  Detail & Related papers  (2020-11-23T13:35:03Z)
• Distilling Localization for Self-Supervised Representation Learning [82.79808902674282]
  Contrastive learning has revolutionized unsupervised representation learning. Current contrastive models are ineffective at localizing the foreground object. We propose a data-driven approach for learning in variance to backgrounds.
  arXiv  Detail & Related papers  (2020-04-14T16:29:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.