Bokeh Rendering Based on Adaptive Depth Calibration Network
- URL: http://arxiv.org/abs/2302.10808v1
- Date: Tue, 21 Feb 2023 16:33:51 GMT
- Title: Bokeh Rendering Based on Adaptive Depth Calibration Network
- Authors: Lu Liu, Lei Zhou, Yuhan Dong
- Abstract summary: Bokeh rendering is a popular technique used in photography to create an aesthetically pleasing effect.
Mobile phones are not able to capture natural shallow depth-of-field photos.
We propose a novel method for bokeh rendering using the Vision Transformer, a recent and powerful deep learning architecture.
- Score: 13.537088629080122
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Bokeh rendering is a popular and effective technique used in photography to
create an aesthetically pleasing effect. It is widely used to blur the
background and highlight the subject in the foreground, thereby drawing the
viewer's attention to the main focus of the image. In traditional digital
single-lens reflex cameras (DSLRs), this effect is achieved through the use of
a large aperture lens. This allows the camera to capture images with shallow
depth-of-field, in which only a small area of the image is in sharp focus,
while the rest of the image is blurred. However, the hardware embedded in
mobile phones is typically much smaller and more limited than that found in
DSLRs. Consequently, mobile phones are not able to capture natural shallow
depth-of-field photos, which can be a significant limitation for mobile
photography. To address this challenge, in this paper, we propose a novel
method for bokeh rendering using the Vision Transformer, a recent and powerful
deep learning architecture. Our approach employs an adaptive depth calibration
network that acts as a confidence level to compensate for errors in monocular
depth estimation. This network is used to supervise the rendering process in
conjunction with depth information, allowing for the generation of high-quality
bokeh images at high resolutions. Our experiments demonstrate that our proposed
method outperforms state-of-the-art methods, achieving about 24.7% improvements
on LPIPS and obtaining higher PSNR scores.
Related papers
- Realistic Bokeh Effect Rendering on Mobile GPUs, Mobile AI & AIM 2022
challenge: Report [75.79829464552311]
This challenge was to develop an efficient end-to-end AI-based rendering approach that can run on modern smartphone models.
The resulting model was evaluated on the Kirin 9000's Mali GPU that provides excellent acceleration results for the majority of common deep learning ops.
arXiv Detail & Related papers (2022-11-07T22:42:02Z) - Bokeh-Loss GAN: Multi-Stage Adversarial Training for Realistic
Edge-Aware Bokeh [3.8811606213997587]
We tackle the problem of monocular bokeh synthesis, where we attempt to render a shallow depth of field image from a single all-in-focus image.
Unlike in DSLR cameras, this effect can not be captured directly in mobile cameras due to the physical constraints of the mobile aperture.
We propose a network-based approach that is capable of rendering realistic monocular bokeh from single image inputs.
arXiv Detail & Related papers (2022-08-25T20:57:07Z) - Direct Handheld Burst Imaging to Simulated Defocus [1.7403133838762446]
A shallow depth-of-field image keeps the subject in focus, and the foreground and background contexts blurred.
We present a learning-based method to synthesize the defocus blur in shallow depth-of-field images from handheld bursts.
Our method does not suffer from artifacts due to inaccurate or ambiguous depth estimation, and it is well-suited to portrait photography.
arXiv Detail & Related papers (2022-07-09T01:59:36Z) - Single image deep defocus estimation and its applications [82.93345261434943]
We train a deep neural network to classify image patches into one of the 20 levels of blurriness.
The trained model is used to determine the patch blurriness which is then refined by applying an iterative weighted guided filter.
The result is a defocus map that carries the information of the degree of blurriness for each pixel.
arXiv Detail & Related papers (2021-07-30T06:18:16Z) - Stacked Deep Multi-Scale Hierarchical Network for Fast Bokeh Effect
Rendering from a Single Image [9.010261475120627]
The Bokeh Effect is one of the most desirable effects in photography for rendering artistic and aesthetic photos.
In this paper, we used an end-to-end Deep Multi-Scale Hierarchical Network (DMSHN) model for direct Bokeh effect rendering of images captured from the monocular camera.
Stacked DMSHN achieves state-of-the-art results on a large scale EBB! dataset with around 6x less runtime compared to the current state-of-the-art model in processing HD quality images.
arXiv Detail & Related papers (2021-05-15T08:45:20Z) - AIM 2020 Challenge on Rendering Realistic Bokeh [95.87775182820518]
This paper reviews the second AIM realistic bokeh effect rendering challenge.
The goal was to learn a realistic shallow focus technique using a large-scale EBB! bokeh dataset.
The participants had to render bokeh effect based on only one single frame without any additional data from other cameras or sensors.
arXiv Detail & Related papers (2020-11-10T09:15:38Z) - Deep Photo Cropper and Enhancer [65.11910918427296]
We propose a new type of image enhancement problem: to crop an image which is embedded within a photo.
We split our proposed approach into two deep networks: deep photo cropper and deep image enhancer.
In the photo cropper network, we employ a spatial transformer to extract the embedded image.
In the photo enhancer, we employ super-resolution to increase the number of pixels in the embedded image.
arXiv Detail & Related papers (2020-08-03T03:50:20Z) - Defocus Blur Detection via Depth Distillation [64.78779830554731]
We introduce depth information into DBD for the first time.
In detail, we learn the defocus blur from ground truth and the depth distilled from a well-trained depth estimation network.
Our approach outperforms 11 other state-of-the-art methods on two popular datasets.
arXiv Detail & Related papers (2020-07-16T04:58:09Z) - Rendering Natural Camera Bokeh Effect with Deep Learning [95.86933125733673]
Bokeh is an important artistic effect used to highlight the main object of interest on the photo.
Mobile cameras are unable to produce shallow depth-of-field photos due to a very small aperture diameter of their optics.
We propose to learn a realistic shallow focus technique directly from the photos produced by DSLR cameras.
arXiv Detail & Related papers (2020-06-10T07:28:06Z) - Depth-aware Blending of Smoothed Images for Bokeh Effect Generation [10.790210744021072]
In this paper, an end-to-end deep learning framework is proposed to generate high-quality bokeh effect from images.
The network is lightweight and can process an HD image in 0.03 seconds.
This approach ranked second in AIM 2019 Bokeh effect challenge-Perceptual Track.
arXiv Detail & Related papers (2020-05-28T18:11:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.