Weak-shot Semantic Segmentation by Transferring Semantic Affinity and Boundary

• URL: http://arxiv.org/abs/2110.01519v1
• Date: Mon, 4 Oct 2021 15:37:25 GMT
• Title: Weak-shot Semantic Segmentation by Transferring Semantic Affinity and Boundary
• Authors: Siyuan Zhou and Li Niu and Jianlou Si and Chen Qian and Liqing Zhang
• Abstract summary: We show that existing fully-annotated base categories can help segment objects of novel categories with only image-level labels. We propose a method under the WSSS framework to transfer semantic affinity and boundary from base categories to novel ones.
• Score: 23.331708585468814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Weakly-supervised semantic segmentation (WSSS) with image-level labels has been widely studied to relieve the annotation burden of the traditional segmentation task. In this paper, we show that existing fully-annotated base categories can help segment objects of novel categories with only image-level labels, even if base and novel categories have no overlap. We refer to this task as weak-shot semantic segmentation, which could also be treated as WSSS with auxiliary fully-annotated categories. Recent advanced WSSS methods usually obtain class activation maps (CAMs) and refine them by affinity propagation. Based on the observation that semantic affinity and boundary are class-agnostic, we propose a method under the WSSS framework to transfer semantic affinity and boundary from base categories to novel ones. As a result, we find that pixel-level annotation of base categories can facilitate affinity learning and propagation, leading to higher-quality CAMs of novel categories. Extensive experiments on PASCAL VOC 2012 dataset demonstrate that our method significantly outperforms WSSS baselines on novel categories.

Related papers

• Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised Semantic Segmentation [79.05949524349005]
  We propose AuxSegNet+, a weakly supervised auxiliary learning framework to explore the rich information from saliency maps. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps.
  arXiv  Detail & Related papers  (2024-03-02T10:03:21Z)
• Contrastive Bootstrapping for Label Refinement [34.55195008779178]
  We propose a lightweight contrastive clustering-based bootstrapping method to iteratively refine the labels of passages. Experiments on NYT and 20News show that our method outperforms the state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2023-06-07T15:49:04Z)
• Advancing Incremental Few-shot Semantic Segmentation via Semantic-guided Relation Alignment and Adaptation [98.51938442785179]
  Incremental few-shot semantic segmentation aims to incrementally extend a semantic segmentation model to novel classes. This task faces a severe semantic-aliasing issue between base and novel classes due to data imbalance. We propose the Semantic-guided Relation Alignment and Adaptation (SRAA) method that fully considers the guidance of prior semantic information.
  arXiv  Detail & Related papers  (2023-05-18T10:40:52Z)
• SLAM: Semantic Learning based Activation Map for Weakly Supervised Semantic Segmentation [34.996841532954925]
  We propose a novel semantic learning based framework for WSSS, named SLAM (Semantic Learning based Activation Map) We firstly design a semantic encoder to learn semantics of each object category and extract category-specific semantic embeddings from an input image. Four loss functions, i.e., category-foreground, category-background, activation regularization, and consistency loss are proposed to ensure the correctness, completeness, compactness and consistency of the activation map.
  arXiv  Detail & Related papers  (2022-10-22T11:17:30Z)
• MuSCLe: A Multi-Strategy Contrastive Learning Framework for Weakly Supervised Semantic Segmentation [39.858844102571176]
  Weakly supervised semantic segmentation (WSSS) relies on weak labels such as image level annotations rather than pixel level annotations required by supervised semantic segmentation (SSS) methods. We propose a novel Multi-Strategy Contrastive Learning (MuSCLe) framework to obtain enhanced feature representations and improve WSSS performance.
  arXiv  Detail & Related papers  (2022-01-18T14:38:50Z)
• Novel Class Discovery in Semantic Segmentation [104.30729847367104]
  We introduce a new setting of Novel Class Discovery in Semantic (NCDSS) It aims at segmenting unlabeled images containing new classes given prior knowledge from a labeled set of disjoint classes. In NCDSS, we need to distinguish the objects and background, and to handle the existence of multiple classes within an image. We propose the Entropy-based Uncertainty Modeling and Self-training (EUMS) framework to overcome noisy pseudo-labels.
  arXiv  Detail & Related papers  (2021-12-03T13:31:59Z)
• Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation [88.49669148290306]
  We propose a novel weakly supervised multi-task framework called AuxSegNet to leverage saliency detection and multi-label image classification as auxiliary tasks. Inspired by their similar structured semantics, we also propose to learn a cross-task global pixel-level affinity map from the saliency and segmentation representations. The learned cross-task affinity can be used to refine saliency predictions and propagate CAM maps to provide improved pseudo labels for both tasks.
  arXiv  Detail & Related papers  (2021-07-25T11:39:58Z)
• Towards Novel Target Discovery Through Open-Set Domain Adaptation [73.81537683043206]
  Open-set domain adaptation (OSDA) considers that the target domain contains samples from novel categories unobserved in external source domain. We propose a novel framework to accurately identify the seen categories in target domain, and effectively recover the semantic attributes for unseen categories.
  arXiv  Detail & Related papers  (2021-05-06T04:22:29Z)
• Domain Adaptive Semantic Segmentation Using Weak Labels [115.16029641181669]
  We propose a novel framework for domain adaptation in semantic segmentation with image-level weak labels in the target domain. We develop a weak-label classification module to enforce the network to attend to certain categories. In experiments, we show considerable improvements with respect to the existing state-of-the-arts in UDA and present a new benchmark in the WDA setting.
  arXiv  Detail & Related papers  (2020-07-30T01:33:57Z)
• Joint Embedding of Words and Category Labels for Hierarchical Multi-label Text Classification [4.2750700546937335]
  hierarchical text classification (HTC) has received extensive attention and has broad application prospects. We propose a joint embedding of text and parent category based on hierarchical fine-tuning ordered neurons LSTM (HFT-ONLSTM) for HTC.
  arXiv  Detail & Related papers  (2020-04-06T11:06:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.