S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation

• URL: http://arxiv.org/abs/2104.00877v1
• Date: Fri, 2 Apr 2021 03:55:41 GMT
• Title: S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation
• Authors: Xiaotian Chen, Yuwang Wang, Xuejin Chen, Wenjun Zeng
• Abstract summary: Human can infer the 3D geometry of a scene from a sketch instead of a realistic image, which indicates that the spatial structure plays a fundamental role in understanding the depth of scenes. We are the first to explore the learning of a depth-specific structural representation, which captures the essential feature for depth estimation and ignores irrelevant style information. Our S2R-DepthNet can be well generalized to unseen real-world data directly even though it is only trained on synthetic data.
• Score: 63.58891781246175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human can infer the 3D geometry of a scene from a sketch instead of a realistic image, which indicates that the spatial structure plays a fundamental role in understanding the depth of scenes. We are the first to explore the learning of a depth-specific structural representation, which captures the essential feature for depth estimation and ignores irrelevant style information. Our S2R-DepthNet (Synthetic to Real DepthNet) can be well generalized to unseen real-world data directly even though it is only trained on synthetic data. S2R-DepthNet consists of: a) a Structure Extraction (STE) module which extracts a domaininvariant structural representation from an image by disentangling the image into domain-invariant structure and domain-specific style components, b) a Depth-specific Attention (DSA) module, which learns task-specific knowledge to suppress depth-irrelevant structures for better depth estimation and generalization, and c) a depth prediction module (DP) to predict depth from the depth-specific representation. Without access of any real-world images, our method even outperforms the state-of-the-art unsupervised domain adaptation methods which use real-world images of the target domain for training. In addition, when using a small amount of labeled real-world data, we achieve the state-ofthe-art performance under the semi-supervised setting.

Related papers

• Depth-guided Texture Diffusion for Image Semantic Segmentation [47.46257473475867]
  We introduce a Depth-guided Texture Diffusion approach that effectively tackles the outlined challenge. Our method extracts low-level features from edges and textures to create a texture image. By integrating this enriched depth map with the original RGB image into a joint feature embedding, our method effectively bridges the disparity between the depth map and the image.
  arXiv  Detail & Related papers  (2024-08-17T04:55:03Z)
• Depth-aware Volume Attention for Texture-less Stereo Matching [67.46404479356896]
  We propose a lightweight volume refinement scheme to tackle the texture deterioration in practical outdoor scenarios. We introduce a depth volume supervised by the ground-truth depth map, capturing the relative hierarchy of image texture. Local fine structure and context are emphasized to mitigate ambiguity and redundancy during volume aggregation.
  arXiv  Detail & Related papers  (2024-02-14T04:07:44Z)
• Transferring to Real-World Layouts: A Depth-aware Framework for Scene Adaptation [34.786268652516355]
  Scene segmentation via unsupervised domain adaptation (UDA) enables the transfer of knowledge acquired from source synthetic data to real-world target data. We propose a depth-aware framework to explicitly leverage depth estimation to mix the categories and facilitate the two complementary tasks, i.e., segmentation and depth learning. In particular, the framework contains a Depth-guided Contextual Filter (DCF) forndata augmentation and a cross-task encoder for contextual learning.
  arXiv  Detail & Related papers  (2023-11-21T15:39:21Z)
• RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion [31.70022495622075]
  We explore a repetitive design in our image guided network to gradually and sufficiently recover depth values. In the former branch, we design a dense repetitive hourglass network (DRHN) to extract discriminative image features of complex environments. In the latter branch, we present a repetitive guidance (RG) module based on dynamic convolution, in which an efficient convolution factorization is proposed to reduce the complexity. In addition, we propose a region-aware spatial propagation network (RASPN) for further depth refinement based on the semantic prior constraint.
  arXiv  Detail & Related papers  (2023-09-01T09:11:20Z)
• Source-free Depth for Object Pop-out [113.24407776545652]
  Modern learning-based methods offer promising depth maps by inference in the wild. We adapt such depth inference models for object segmentation using the objects' "pop-out" prior in 3D. Our experiments on eight datasets consistently demonstrate the benefit of our method in terms of both performance and generalizability.
  arXiv  Detail & Related papers  (2022-12-10T21:57:11Z)
• Self-Guided Instance-Aware Network for Depth Completion and Enhancement [6.319531161477912]
  Existing methods directly interpolate the missing depth measurements based on pixel-wise image content and the corresponding neighboring depth values. We propose a novel self-guided instance-aware network (SG-IANet) that utilize self-guided mechanism to extract instance-level features that is needed for depth restoration.
  arXiv  Detail & Related papers  (2021-05-25T19:41:38Z)
• Learning Depth With Very Sparse Supervision [57.911425589947314]
  This paper explores the idea that perception gets coupled to 3D properties of the world via interaction with the environment. We train a specialized global-local network architecture with what would be available to a robot interacting with the environment. Experiments on several datasets show that, when ground truth is available even for just one of the image pixels, the proposed network can learn monocular dense depth estimation up to 22.5% more accurately than state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-03-02T10:44:13Z)
• Shallow2Deep: Indoor Scene Modeling by Single Image Understanding [42.87957414916607]
  We present an automatic indoor scene modeling approach using deep features from neural networks. Given a single RGB image, our method simultaneously recovers semantic contents, 3D geometry and object relationship.
  arXiv  Detail & Related papers  (2020-02-22T23:27:22Z)
• Single Image Depth Estimation Trained via Depth from Defocus Cues [105.67073923825842]
  Estimating depth from a single RGB image is a fundamental task in computer vision. In this work, we rely, instead of different views, on depth from focus cues. We present results that are on par with supervised methods on KITTI and Make3D datasets and outperform unsupervised learning approaches.
  arXiv  Detail & Related papers  (2020-01-14T20:22:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.