RGBD-Net: Predicting color and depth images for novel views synthesis

• URL: http://arxiv.org/abs/2011.14398v1
• Date: Sun, 29 Nov 2020 16:42:53 GMT
• Title: RGBD-Net: Predicting color and depth images for novel views synthesis
• Authors: Phong Nguyen, Animesh Karnewar, Lam Huynh, Esa Rahtu, Jiri Matas, Janne Heikkila
• Abstract summary: RGBD-Net is proposed to predict the depth map and the color images at the target pose in a multi-scale manner. The results indicate that RGBD-Net generalizes well to previously unseen data.
• Score: 46.233701784858184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the problem of novel view synthesis from an unstructured set of reference images. A new method called RGBD-Net is proposed to predict the depth map and the color images at the target pose in a multi-scale manner. The reference views are warped to the target pose to obtain multi-scale plane sweep volumes, which are then passed to our first module, a hierarchical depth regression network which predicts the depth map of the novel view. Second, a depth-aware generator network refines the warped novel views and renders the final target image. These two networks can be trained with or without depth supervision. In experimental evaluation, RGBD-Net not only produces novel views with higher quality than the previous state-of-the-art methods, but also the obtained depth maps enable reconstruction of more accurate 3D point clouds than the existing multi-view stereo methods. The results indicate that RGBD-Net generalizes well to previously unseen data.

Related papers

• AGG-Net: Attention Guided Gated-convolutional Network for Depth Image Completion [1.8820731605557168]
  We propose a new model for depth image completion based on the Attention Guided Gated-convolutional Network (AGG-Net) In the encoding stage, an Attention Guided Gated-Convolution (AG-GConv) module is proposed to realize the fusion of depth and color features at different scales. In the decoding stage, an Attention Guided Skip Connection (AG-SC) module is presented to avoid introducing too many depth-irrelevant features to the reconstruction.
  arXiv  Detail & Related papers  (2023-09-04T14:16:08Z)
• Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images [15.614631883233898]
  Existing methods have shown promising results leveraging monocular depth estimation and color inpainting with layered depth representations. We propose a new method based on the multiplane image (MPI) representation. The experiments on both synthetic and real datasets demonstrate that our trained model works remarkably well and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2022-05-24T02:57:16Z)
• NVS-MonoDepth: Improving Monocular Depth Prediction with Novel View Synthesis [74.4983052902396]
  We propose a novel training method split in three main steps to improve monocular depth estimation. Experimental results prove that our method achieves state-of-the-art or comparable performance on the KITTI and NYU-Depth-v2 datasets.
  arXiv  Detail & Related papers  (2021-12-22T12:21:08Z)
• VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction [71.83308989022635]
  In this paper, we advocate that replicating the traditional two stages framework with deep neural networks improves both the interpretability and the accuracy of the results. Our network operates in two steps: 1) the local computation of the local depth maps with a deep MVS technique, and, 2) the depth maps and images' features fusion to build a single TSDF volume. In order to improve the matching performance between images acquired from very different viewpoints, we introduce a rotation-invariant 3D convolution kernel called PosedConv.
  arXiv  Detail & Related papers  (2021-08-19T11:33:58Z)
• Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion [56.85837052421469]
  Estimating scene geometry from data obtained with cost-effective sensors is key for robots and self-driving cars. In this paper, we study the problem of predicting dense depth from a single RGB image with optional sparse measurements from low-cost active depth sensors. We introduce Sparse Networks (SANs), a new module enabling monodepth networks to perform both the tasks of depth prediction and completion.
  arXiv  Detail & Related papers  (2021-03-30T21:22:26Z)
• Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
  We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object. We first estimate per-view depth maps using a deep multi-view stereo network. These depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture.
  arXiv  Detail & Related papers  (2020-03-27T21:28:54Z)
• 3dDepthNet: Point Cloud Guided Depth Completion Network for Sparse Depth and Single Color Image [42.13930269841654]
  Our network offers a novel 3D-to-2D coarse-to-fine dual densification design that is both accurate and lightweight. Experiments on the KITTI dataset show our network achieves state-of-art accuracy while being more efficient.
  arXiv  Detail & Related papers  (2020-03-20T10:19:32Z)
• Depth Completion Using a View-constrained Deep Prior [73.21559000917554]
  Recent work has shown that the structure of convolutional neural networks (CNNs) induces a strong prior that favors natural images. This prior, known as a deep image prior (DIP), is an effective regularizer in inverse problems such as image denoising and inpainting. We extend the concept of the DIP to depth images. Given color images and noisy and incomplete target depth maps, we reconstruct a depth map restored by virtue of using the CNN network structure as a prior.
  arXiv  Detail & Related papers  (2020-01-21T21:56:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.