S3Net: A Single Stream Structure for Depth Guided Image Relighting
- URL: http://arxiv.org/abs/2105.00681v2
- Date: Wed, 5 May 2021 02:09:53 GMT
- Title: S3Net: A Single Stream Structure for Depth Guided Image Relighting
- Authors: Hao-Hsiang Yang and Wei-Ting Chen and and Sy-Yen Kuo
- Abstract summary: We propose a deep learning-based neural Single Stream Structure network called S3Net for depth guided image relighting.
Experiments performed on challenging benchmark show that the proposed model achieves the 3 rd highest SSIM in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge.
- Score: 13.201978111555817
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Depth guided any-to-any image relighting aims to generate a relit image from
the original image and corresponding depth maps to match the illumination
setting of the given guided image and its depth map. To the best of our
knowledge, this task is a new challenge that has not been addressed in the
previous literature. To address this issue, we propose a deep learning-based
neural Single Stream Structure network called S3Net for depth guided image
relighting. This network is an encoder-decoder model. We concatenate all images
and corresponding depth maps as the input and feed them into the model. The
decoder part contains the attention module and the enhanced module to focus on
the relighting-related regions in the guided images. Experiments performed on
challenging benchmark show that the proposed model achieves the 3 rd highest
SSIM in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge.
Related papers
- RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth
Completion [31.70022495622075]
We explore a repetitive design in our image guided network to gradually and sufficiently recover depth values.
In the former branch, we design a dense repetitive hourglass network (DRHN) to extract discriminative image features of complex environments.
In the latter branch, we present a repetitive guidance (RG) module based on dynamic convolution, in which an efficient convolution factorization is proposed to reduce the complexity.
In addition, we propose a region-aware spatial propagation network (RASPN) for further depth refinement based on the semantic prior constraint.
arXiv Detail & Related papers (2023-09-01T09:11:20Z) - Facial Depth and Normal Estimation using Single Dual-Pixel Camera [81.02680586859105]
We introduce a DP-oriented Depth/Normal network that reconstructs the 3D facial geometry.
It contains the corresponding ground-truth 3D models including depth map and surface normal in metric scale.
It achieves state-of-the-art performances over recent DP-based depth/normal estimation methods.
arXiv Detail & Related papers (2021-11-25T05:59:27Z) - SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware
Inpainting [54.419266357283966]
Single image 3D photography enables viewers to view a still image from novel viewpoints.
Recent approaches combine monocular depth networks with inpainting networks to achieve compelling results.
We present SLIDE, a modular and unified system for single image 3D photography.
arXiv Detail & Related papers (2021-09-02T16:37:20Z) - RigNet: Repetitive Image Guided Network for Depth Completion [20.66405067066299]
Recent approaches mainly focus on image guided learning to predict dense results.
blurry image guidance and object structures in depth still impede the performance of image guided frameworks.
We explore a repetitive design in our image guided network to sufficiently and gradually recover depth values.
Our method achieves state-of-the-art result on the NYUv2 dataset and ranks 1st on the KITTI benchmark at the time of submission.
arXiv Detail & Related papers (2021-07-29T08:00:33Z) - Multi-modal Bifurcated Network for Depth Guided Image Relighting [13.857410735989301]
We propose a deep learning-based method called multi-modal bifurcated network (MBNet) for depth guided image relighting.
This model extracts the image and the depth features by the bifurcated network in the encoder.
Experiments conducted on the VIDIT dataset show that the proposed solution obtains the textbf1$st$ place in terms of SSIM and PMS.
arXiv Detail & Related papers (2021-05-03T08:52:25Z) - Memory-Augmented Reinforcement Learning for Image-Goal Navigation [67.3963444878746]
We present a novel method that leverages a cross-episode memory to learn to navigate.
In order to avoid overfitting, we propose to use data augmentation on the RGB input during training.
We obtain this competitive performance from RGB input only, without access to additional sensors such as position or depth.
arXiv Detail & Related papers (2021-01-13T16:30:20Z) - RGBD-Net: Predicting color and depth images for novel views synthesis [46.233701784858184]
RGBD-Net is proposed to predict the depth map and the color images at the target pose in a multi-scale manner.
The results indicate that RGBD-Net generalizes well to previously unseen data.
arXiv Detail & Related papers (2020-11-29T16:42:53Z) - Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object.
We first estimate per-view depth maps using a deep multi-view stereo network.
These depth maps are used to coarsely align the different views.
We propose a novel multi-view reflectance estimation network architecture.
arXiv Detail & Related papers (2020-03-27T21:28:54Z) - 3dDepthNet: Point Cloud Guided Depth Completion Network for Sparse Depth
and Single Color Image [42.13930269841654]
Our network offers a novel 3D-to-2D coarse-to-fine dual densification design that is both accurate and lightweight.
Experiments on the KITTI dataset show our network achieves state-of-art accuracy while being more efficient.
arXiv Detail & Related papers (2020-03-20T10:19:32Z) - Depth Completion Using a View-constrained Deep Prior [73.21559000917554]
Recent work has shown that the structure of convolutional neural networks (CNNs) induces a strong prior that favors natural images.
This prior, known as a deep image prior (DIP), is an effective regularizer in inverse problems such as image denoising and inpainting.
We extend the concept of the DIP to depth images. Given color images and noisy and incomplete target depth maps, we reconstruct a depth map restored by virtue of using the CNN network structure as a prior.
arXiv Detail & Related papers (2020-01-21T21:56:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.