SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion

• URL: http://arxiv.org/abs/2209.00381v2
• Date: Wed, 6 Mar 2024 12:52:22 GMT
• Title: SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion
• Authors: Juan Pablo Lagos and Esa Rahtu
• Abstract summary: We propose a new end-to-end model for performing semantic segmentation and depth completion jointly. Our approach relies on RGB and sparse depth as inputs to our model and produces a dense depth map and the corresponding semantic segmentation image. Experiments done on Virtual KITTI 2 dataset, demonstrate and provide further evidence, that combining both tasks, semantic segmentation and depth completion, in a multi-task network can effectively improve the performance of each task.
• Score: 18.19171031755595
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Holistic scene understanding is pivotal for the performance of autonomous machines. In this paper we propose a new end-to-end model for performing semantic segmentation and depth completion jointly. The vast majority of recent approaches have developed semantic segmentation and depth completion as independent tasks. Our approach relies on RGB and sparse depth as inputs to our model and produces a dense depth map and the corresponding semantic segmentation image. It consists of a feature extractor, a depth completion branch, a semantic segmentation branch and a joint branch which further processes semantic and depth information altogether. The experiments done on Virtual KITTI 2 dataset, demonstrate and provide further evidence, that combining both tasks, semantic segmentation and depth completion, in a multi-task network can effectively improve the performance of each task. Code is available at https://github.com/juanb09111/semantic depth.

Related papers

• Joint Depth Prediction and Semantic Segmentation with Multi-View SAM [59.99496827912684]
  We propose a Multi-View Stereo (MVS) technique for depth prediction that benefits from rich semantic features of the Segment Anything Model (SAM) This enhanced depth prediction, in turn, serves as a prompt to our Transformer-based semantic segmentation decoder.
  arXiv  Detail & Related papers  (2023-10-31T20:15:40Z)
• PanDepth: Joint Panoptic Segmentation and Depth Completion [19.642115764441016]
  We propose a multi-task model for panoptic segmentation and depth completion using RGB images and sparse depth maps. Our model successfully predicts fully dense depth maps and performs semantic segmentation, instance segmentation, and panoptic segmentation for every input frame.
  arXiv  Detail & Related papers  (2022-12-29T05:37:38Z)
• Semantics-Depth-Symbiosis: Deeply Coupled Semi-Supervised Learning of Semantics and Depth [83.94528876742096]
  We tackle the MTL problem of two dense tasks, ie, semantic segmentation and depth estimation, and present a novel attention module called Cross-Channel Attention Module (CCAM) In a true symbiotic spirit, we then formulate a novel data augmentation for the semantic segmentation task using predicted depth called AffineMix, and a simple depth augmentation using predicted semantics called ColorAug. Finally, we validate the performance gain of the proposed method on the Cityscapes dataset, which helps us achieve state-of-the-art results for a semi-supervised joint model based on depth and semantic
  arXiv  Detail & Related papers  (2022-06-21T17:40:55Z)
• PanopticDepth: A Unified Framework for Depth-aware Panoptic Segmentation [41.85216306978024]
  We propose a unified framework for depth-aware panoptic segmentation (DPS) We generate instance-specific kernels to predict depth and segmentation masks for each instance. We add additional instance-level depth cues to assist with supervising the depth learning via a new depth loss.
  arXiv  Detail & Related papers  (2022-06-01T13:00:49Z)
• Improving Semi-Supervised and Domain-Adaptive Semantic Segmentation with Self-Supervised Depth Estimation [94.16816278191477]
  We present a framework for semi-adaptive and domain-supervised semantic segmentation. It is enhanced by self-supervised monocular depth estimation trained only on unlabeled image sequences. We validate the proposed model on the Cityscapes dataset.
  arXiv  Detail & Related papers  (2021-08-28T01:33:38Z)
• Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation [84.34227665232281]
  Domain adaptation for semantic segmentation aims to improve the model performance in the presence of a distribution shift between source and target domain. We leverage the guidance from self-supervised depth estimation, which is available on both domains, to bridge the domain gap. We demonstrate the effectiveness of our proposed approach on the benchmark tasks SYNTHIA-to-Cityscapes and GTA-to-Cityscapes.
  arXiv  Detail & Related papers  (2021-04-28T07:47:36Z)
• SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular images [94.36401543589523]
  We introduce the concept of semantic objectness to exploit the geometric relationship of these two tasks. We then propose a Semantic Object and Depth Estimation Network (SOSD-Net) based on the objectness assumption. To the best of our knowledge, SOSD-Net is the first network that exploits the geometry constraint for simultaneous monocular depth estimation and semantic segmentation.
  arXiv  Detail & Related papers  (2021-01-19T02:41:03Z)
• Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation [90.87105131054419]
  We present a framework for semi-supervised semantic segmentation, which is enhanced by self-supervised monocular depth estimation from unlabeled image sequences. We validate the proposed model on the Cityscapes dataset, where all three modules demonstrate significant performance gains.
  arXiv  Detail & Related papers  (2020-12-19T21:18:03Z)
• Multi-task GANs for Semantic Segmentation and Depth Completion with Cycle Consistency [7.273142068778457]
  We propose multi-task generative adversarial networks (Multi-task GANs), which are competent in semantic segmentation and depth completion. In this paper, we improve the details of generated semantic images based on CycleGAN by introducing multi-scale spatial pooling blocks and the structural similarity reconstruction loss. Experiments on Cityscapes dataset and KITTI depth completion benchmark show that the Multi-task GANs are capable of achieving competitive performance.
  arXiv  Detail & Related papers  (2020-11-29T04:12:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.