Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

• URL: http://arxiv.org/abs/2104.05833v1
• Date: Mon, 12 Apr 2021 21:41:25 GMT
• Title: Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
• Authors: Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, Sanja Fidler
• Abstract summary: We propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. We learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization.
• Score: 112.68171734288237
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human annotation efforts. This is the goal of semi-supervised learning, which exploits more widely available unlabeled data to complement small labeled data sets. In this paper, we propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. Concretely, we learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images supplemented with only few labeled ones. We build our architecture on top of StyleGAN2, augmented with a label synthesis branch. Image labeling at test time is achieved by first embedding the target image into the joint latent space via an encoder network and test-time optimization, and then generating the label from the inferred embedding. We evaluate our approach in two important domains: medical image segmentation and part-based face segmentation. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization, such as transferring from CT to MRI in medical imaging, and photographs of real faces to paintings, sculptures, and even cartoons and animal faces. Project Page: \url{https://nv-tlabs.github.io/semanticGAN/}

Related papers

• Microscopy Image Segmentation via Point and Shape Regularized Data Synthesis [9.47802391546853]
  We develop a unified pipeline for microscopy image segmentation using synthetically generated training data. Our framework achieves comparable results to models trained on authentic microscopy images with dense labels.
  arXiv  Detail & Related papers  (2023-08-18T22:00:53Z)
• A Semi-Paired Approach For Label-to-Image Translation [6.888253564585197]
  We introduce the first semi-supervised (semi-paired) framework for label-to-image translation. In the semi-paired setting, the model has access to a small set of paired data and a larger set of unpaired images and labels. We propose a training algorithm for this shared network, and we present a rare classes sampling algorithm to focus on under-represented classes.
  arXiv  Detail & Related papers  (2023-06-23T16:13:43Z)
• Learning to Annotate Part Segmentation with Gradient Matching [58.100715754135685]
  This paper focuses on tackling semi-supervised part segmentation tasks by generating high-quality images with a pre-trained GAN. In particular, we formulate the annotator learning as a learning-to-learn problem. We show that our method can learn annotators from a broad range of labelled images including real images, generated images, and even analytically rendered images.
  arXiv  Detail & Related papers  (2022-11-06T01:29:22Z)
• Maximize the Exploration of Congeneric Semantics for Weakly Supervised Semantic Segmentation [27.155133686127474]
  We construct a graph neural network (P-GNN) based on the self-detected patches from different images that contain the same class labels. We conduct experiments on the popular PASCAL VOC 2012 benchmarks, and our model yields state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-10-08T08:59:16Z)
• Semi-weakly Supervised Contrastive Representation Learning for Retinal Fundus Images [0.2538209532048867]
  We propose a semi-weakly supervised contrastive learning framework for representation learning using semi-weakly annotated images. We empirically validate the transfer learning performance of SWCL on seven public retinal fundus datasets.
  arXiv  Detail & Related papers  (2021-08-04T15:50:09Z)
• Open-World Entity Segmentation [70.41548013910402]
  We introduce a new image segmentation task, termed Entity (ES) with the aim to segment all visual entities in an image without considering semantic category labels. All semantically-meaningful segments are equally treated as categoryless entities and there is no thing-stuff distinction. ES enables the following: (1) merging multiple datasets to form a large training set without the need to resolve label conflicts; (2) any model trained on one dataset can generalize exceptionally well to other datasets with unseen domains.
  arXiv  Detail & Related papers  (2021-07-29T17:59:05Z)
• Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation [49.90178055521207]
  This work addresses weakly supervised semantic segmentation (WSSS), with the goal of bridging the gap between image-level annotations and pixel-level segmentation. We formulate WSSS as a novel group-wise learning task that explicitly models semantic dependencies in a group of images to estimate more reliable pseudo ground-truths. In particular, we devise a graph neural network (GNN) for group-wise semantic mining, wherein input images are represented as graph nodes.
  arXiv  Detail & Related papers  (2020-12-09T12:40:13Z)
• Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation [128.03739769844736]
  Two neural co-attentions are incorporated into the classifier to capture cross-image semantic similarities and differences. In addition to boosting object pattern learning, the co-attention can leverage context from other related images to improve localization map inference. Our algorithm sets new state-of-the-arts on all these settings, demonstrating well its efficacy and generalizability.
  arXiv  Detail & Related papers  (2020-07-03T21:53:46Z)
• Adversarial Learning for Personalized Tag Recommendation [61.76193196463919]
  We propose an end-to-end deep network which can be trained on large-scale datasets. A joint training of user-preference and visual encoding allows the network to efficiently integrate the visual preference with tagging behavior. We demonstrate the effectiveness of the proposed model on two different large-scale and publicly available datasets.
  arXiv  Detail & Related papers  (2020-04-01T20:41:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.