Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation

• URL: http://arxiv.org/abs/2007.01947v2
• Date: Wed, 8 Jul 2020 11:51:59 GMT
• Title: Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation
• Authors: Guolei Sun and Wenguan Wang and Jifeng Dai and Luc Van Gool
• Abstract summary: Two neural co-attentions are incorporated into the classifier to capture cross-image semantic similarities and differences. In addition to boosting object pattern learning, the co-attention can leverage context from other related images to improve localization map inference. Our algorithm sets new state-of-the-arts on all these settings, demonstrating well its efficacy and generalizability.
• Score: 128.03739769844736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper studies the problem of learning semantic segmentation from image-level supervision only. Current popular solutions leverage object localization maps from classifiers as supervision signals, and struggle to make the localization maps capture more complete object content. Rather than previous efforts that primarily focus on intra-image information, we address the value of cross-image semantic relations for comprehensive object pattern mining. To achieve this, two neural co-attentions are incorporated into the classifier to complimentarily capture cross-image semantic similarities and differences. In particular, given a pair of training images, one co-attention enforces the classifier to recognize the common semantics from co-attentive objects, while the other one, called contrastive co-attention, drives the classifier to identify the unshared semantics from the rest, uncommon objects. This helps the classifier discover more object patterns and better ground semantics in image regions. In addition to boosting object pattern learning, the co-attention can leverage context from other related images to improve localization map inference, hence eventually benefiting semantic segmentation learning. More essentially, our algorithm provides a unified framework that handles well different WSSS settings, i.e., learning WSSS with (1) precise image-level supervision only, (2) extra simple single-label data, and (3) extra noisy web data. It sets new state-of-the-arts on all these settings, demonstrating well its efficacy and generalizability. Moreover, our approach ranked 1st place in the Weakly-Supervised Semantic Segmentation Track of CVPR2020 Learning from Imperfect Data Challenge.

Related papers

• Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised Semantic Segmentation [79.05949524349005]
  We propose AuxSegNet+, a weakly supervised auxiliary learning framework to explore the rich information from saliency maps. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps.
  arXiv  Detail & Related papers  (2024-03-02T10:03:21Z)
• Self-Correlation and Cross-Correlation Learning for Few-Shot Remote Sensing Image Semantic Segmentation [27.59330408178435]
  Few-shot remote sensing semantic segmentation aims at learning to segment target objects from a query image. We propose a Self-Correlation and Cross-Correlation Learning Network for the few-shot remote sensing image semantic segmentation. Our model enhances the generalization by considering both self-correlation and cross-correlation between support and query images.
  arXiv  Detail & Related papers  (2023-09-11T21:53:34Z)
• Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis [139.2216271759332]
  We propose a novel ECGAN for the challenging semantic image synthesis task. The semantic labels do not provide detailed structural information, making it challenging to synthesize local details and structures. The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss. We propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content.
  arXiv  Detail & Related papers  (2023-07-22T14:17:19Z)
• Location-Aware Self-Supervised Transformers [74.76585889813207]
  We propose to pretrain networks for semantic segmentation by predicting the relative location of image parts. We control the difficulty of the task by masking a subset of the reference patch features visible to those of the query. Our experiments show that this location-aware pretraining leads to representations that transfer competitively to several challenging semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-12-05T16:24:29Z)
• Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation [32.76127086403596]
  We propose Contrastive learning for Class-agnostic Activation Map (C$2$AM) generation using unlabeled image data. We form the positive and negative pairs based on the above relations and force the network to disentangle foreground and background. As the network is guided to discriminate cross-image foreground-background, the class-agnostic activation maps learned by our approach generate more complete object regions.
  arXiv  Detail & Related papers  (2022-03-25T08:46:24Z)
• Unsupervised Image Segmentation by Mutual Information Maximization and Adversarial Regularization [7.165364364478119]
  We propose a novel fully unsupervised semantic segmentation method, the so-called Information Maximization and Adrial Regularization (InMARS) Inspired by human perception which parses a scene into perceptual groups, our proposed approach first partitions an input image into meaningful regions (also known as superpixels) Next, it utilizes Mutual-Information-Maximization followed by an adversarial training strategy to cluster these regions into semantically meaningful classes. Our experiments demonstrate that our method achieves the state-of-the-art performance on two commonly used unsupervised semantic segmentation datasets.
  arXiv  Detail & Related papers  (2021-07-01T18:36:27Z)
• Attention-Guided Supervised Contrastive Learning for Semantic Segmentation [16.729068267453897]
  In a per-pixel prediction task, more than one label can exist in a single image for segmentation. We propose an attention-guided supervised contrastive learning approach to highlight a single semantic object every time as the target.
  arXiv  Detail & Related papers  (2021-06-03T05:01:11Z)
• Rethinking of the Image Salient Object Detection: Object-level Semantic Saliency Re-ranking First, Pixel-wise Saliency Refinement Latter [62.26677215668959]
  We propose a lightweight, weakly supervised deep network to coarsely locate semantically salient regions. We then fuse multiple off-the-shelf deep models on these semantically salient regions as the pixel-wise saliency refinement. Our method is simple yet effective, which is the first attempt to consider the salient object detection mainly as an object-level semantic re-ranking problem.
  arXiv  Detail & Related papers  (2020-08-10T07:12:43Z)
• Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis [194.1452124186117]
  We propose a novel ECGAN for the challenging semantic image synthesis task. Our ECGAN achieves significantly better results than state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-03-31T01:23:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.