Improving Semi-Supervised and Domain-Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

• URL: http://arxiv.org/abs/2108.12545v1
• Date: Sat, 28 Aug 2021 01:33:38 GMT
• Title: Improving Semi-Supervised and Domain-Adaptive Semantic Segmentation with Self-Supervised Depth Estimation
• Authors: Lukas Hoyer, Dengxin Dai, Qin Wang, Yuhua Chen, Luc Van Gool
• Abstract summary: We present a framework for semi-adaptive and domain-supervised semantic segmentation. It is enhanced by self-supervised monocular depth estimation trained only on unlabeled image sequences. We validate the proposed model on the Cityscapes dataset.
• Score: 94.16816278191477
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training deep networks for semantic segmentation requires large amounts of labeled training data, which presents a major challenge in practice, as labeling segmentation masks is a highly labor-intensive process. To address this issue, we present a framework for semi-supervised and domain-adaptive semantic segmentation, which is enhanced by self-supervised monocular depth estimation (SDE) trained only on unlabeled image sequences. In particular, we utilize SDE as an auxiliary task comprehensively across the entire learning framework: First, we automatically select the most useful samples to be annotated for semantic segmentation based on the correlation of sample diversity and difficulty between SDE and semantic segmentation. Second, we implement a strong data augmentation by mixing images and labels using the geometry of the scene. Third, we transfer knowledge from features learned during SDE to semantic segmentation by means of transfer and multi-task learning. And fourth, we exploit additional labeled synthetic data with Cross-Domain DepthMix and Matching Geometry Sampling to align synthetic and real data. We validate the proposed model on the Cityscapes dataset, where all four contributions demonstrate significant performance gains, and achieve state-of-the-art results for semi-supervised semantic segmentation as well as for semi-supervised domain adaptation. In particular, with only 1/30 of the Cityscapes labels, our method achieves 92% of the fully-supervised baseline performance and even 97% when exploiting additional data from GTA. The source code is available at https://github.com/lhoyer/improving_segmentation_with_selfsupervised_depth.

Related papers

• Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised Semantic Segmentation [79.05949524349005]
  We propose AuxSegNet+, a weakly supervised auxiliary learning framework to explore the rich information from saliency maps. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps.
  arXiv  Detail & Related papers  (2024-03-02T10:03:21Z)
• Segment Together: A Versatile Paradigm for Semi-Supervised Medical Image Segmentation [17.69933345468061]
  scarcity has become a major obstacle for training powerful deep-learning models for medical image segmentation. We introduce a textbfVersatile textbfSemi-supervised framework to exploit more unlabeled data for semi-supervised medical image segmentation.
  arXiv  Detail & Related papers  (2023-11-20T11:35:52Z)
• Navya3DSeg -- Navya 3D Semantic Segmentation Dataset & split generation for autonomous vehicles [63.20765930558542]
  3D semantic data are useful for core perception tasks such as obstacle detection and ego-vehicle localization. We propose a new dataset, Navya 3D (Navya3DSeg), with a diverse label space corresponding to a large scale production grade operational domain. It contains 23 labeled sequences and 25 supplementary sequences without labels, designed to explore self-supervised and semi-supervised semantic segmentation benchmarks on point clouds.
  arXiv  Detail & Related papers  (2023-02-16T13:41:19Z)
• Training Semantic Segmentation on Heterogeneous Datasets [5.584060970507507]
  We explore semantic segmentation beyond the conventional, single-dataset homogeneous training. We propose a unified framework, that incorporates heterogeneous datasets in a single-network training pipeline. Our framework first curates heterogeneous datasets to bring them into a common format and then trains a single-backbone FCN on all of them simultaneously.
  arXiv  Detail & Related papers  (2023-01-18T16:22:40Z)
• LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds [62.49198183539889]
  We propose a label-efficient semantic segmentation pipeline for outdoor scenes with LiDAR point clouds. Our method co-designs an efficient labeling process with semi/weakly supervised learning. Our proposed method is even highly competitive compared to the fully supervised counterpart with 100% labels.
  arXiv  Detail & Related papers  (2022-10-14T19:13:36Z)
• Threshold-adaptive Unsupervised Focal Loss for Domain Adaptation of Semantic Segmentation [25.626882426111198]
  Unsupervised domain adaptation (UDA) for semantic segmentation has recently gained increasing research attention. In this paper, we propose a novel two-stage entropy-based UDA method for semantic segmentation. Our method achieves state-of-the-art 58.4% and 59.6% mIoUs on SYNTHIA-to-Cityscapes and GTA5-to-Cityscapes using DeepLabV2 and competitive performance using the lightweight BiSeNet.
  arXiv  Detail & Related papers  (2022-08-23T03:48:48Z)
• Scaling up Multi-domain Semantic Segmentation with Sentence Embeddings [81.09026586111811]
  We propose an approach to semantic segmentation that achieves state-of-the-art supervised performance when applied in a zero-shot setting. This is achieved by replacing each class label with a vector-valued embedding of a short paragraph that describes the class. The resulting merged semantic segmentation dataset of over 2 Million images enables training a model that achieves performance equal to that of state-of-the-art supervised methods on 7 benchmark datasets.
  arXiv  Detail & Related papers  (2022-02-04T07:19:09Z)
• Semi-supervised Domain Adaptation based on Dual-level Domain Mixing for Semantic Segmentation [34.790169990156684]
  We focus on a more practical setting of semi-supervised domain adaptation (SSDA) where both a small set of labeled target data and large amounts of labeled source data are available. Two kinds of data mixing methods are proposed to reduce domain gap in both region-level and sample-level respectively. We can obtain two complementary domain-mixed teachers based on dual-level mixed data from holistic and partial views respectively.
  arXiv  Detail & Related papers  (2021-03-08T12:33:17Z)
• Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation [90.87105131054419]
  We present a framework for semi-supervised semantic segmentation, which is enhanced by self-supervised monocular depth estimation from unlabeled image sequences. We validate the proposed model on the Cityscapes dataset, where all three modules demonstrate significant performance gains.
  arXiv  Detail & Related papers  (2020-12-19T21:18:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.