GBSD: Generative Bokeh with Stage Diffusion

• URL: http://arxiv.org/abs/2306.08251v3
• Date: Wed, 17 Apr 2024 03:14:21 GMT
• Title: GBSD: Generative Bokeh with Stage Diffusion
• Authors: Jieren Deng, Xin Zhou, Hao Tian, Zhihong Pan, Derek Aguiar,
• Abstract summary: The bokeh effect is an artistic technique that blurs out-of-focus areas in a photograph. We present GBSD, the first generative text-to-image model that synthesizes photorealistic images with a bokeh style.
• Score: 16.189787907983106
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The bokeh effect is an artistic technique that blurs out-of-focus areas in a photograph and has gained interest due to recent developments in text-to-image synthesis and the ubiquity of smart-phone cameras and photo-sharing apps. Prior work on rendering bokeh effects have focused on post hoc image manipulation to produce similar blurring effects in existing photographs using classical computer graphics or neural rendering techniques, but have either depth discontinuity artifacts or are restricted to reproducing bokeh effects that are present in the training data. More recent diffusion based models can synthesize images with an artistic style, but either require the generation of high-dimensional masks, expensive fine-tuning, or affect global image characteristics. In this paper, we present GBSD, the first generative text-to-image model that synthesizes photorealistic images with a bokeh style. Motivated by how image synthesis occurs progressively in diffusion models, our approach combines latent diffusion models with a 2-stage conditioning algorithm to render bokeh effects on semantically defined objects. Since we can focus the effect on objects, this semantic bokeh effect is more versatile than classical rendering techniques. We evaluate GBSD both quantitatively and qualitatively and demonstrate its ability to be applied in both text-to-image and image-to-image settings.

Related papers

• Variable Aperture Bokeh Rendering via Customized Focal Plane Guidance [18.390543681127976]
  The proposed method has achieved competitive state-of-the-art performance with only 4.4M parameters, which is much lighter than mainstream computational bokeh models. The proposed method has achieved competitive state-of-the-art performance with only 4.4M parameters, which is much lighter than mainstream computational bokeh models.
  arXiv  Detail & Related papers  (2024-10-18T12:04:23Z)
• Curved Diffusion: A Generative Model With Optical Geometry Control [56.24220665691974]
  The influence of different optical systems on the final scene appearance is frequently overlooked. This study introduces a framework that intimately integrates a textto-image diffusion model with the particular lens used in image rendering.
  arXiv  Detail & Related papers  (2023-11-29T13:06:48Z)
• BokehOrNot: Transforming Bokeh Effect with Image Transformer and Lens Metadata Embedding [2.3784282912975345]
  Bokeh effect is an optical phenomenon that offers a pleasant visual experience, typically generated by high-end cameras with wide aperture lenses. We propose a novel universal method for embedding lens metadata into the model and introducing a loss calculation method using alpha masks. Based on the above techniques, we propose the BokehOrNot model, which is capable of producing both blur-to-sharp and sharp-to-blur bokeh effect.
  arXiv  Detail & Related papers  (2023-06-06T21:49:56Z)
• Hybrid Neural Rendering for Large-Scale Scenes with Motion Blur [68.24599239479326]
  We develop a hybrid neural rendering model that makes image-based representation and neural 3D representation join forces to render high-quality, view-consistent images. Our model surpasses state-of-the-art point-based methods for novel view synthesis.
  arXiv  Detail & Related papers  (2023-04-25T08:36:33Z)
• Natural & Adversarial Bokeh Rendering via Circle-of-Confusion Predictive Network [25.319666328268116]
  Bokeh effect is a shallow depth-of-field phenomenon that blurs out-of-focus part in photography. We study a totally new problem, i.e., natural & adversarial bokeh rendering. We propose a hybrid alternative by taking the respective advantages of data-driven and physical-aware methods.
  arXiv  Detail & Related papers  (2021-11-25T09:08:45Z)
• Multi-View Motion Synthesis via Applying Rotated Dual-Pixel Blur Kernels [48.063176079878055]
  One of the primary effects applied to images captured in portrait mode is a synthetic shallow depth of field (DoF) In this work, we follow the trend of rendering the NIMAT effect by introducing a modification on the blur synthesis procedure in portrait mode. Our modification enables a high-quality synthesis of multi-view bokeh from a single image by applying rotated blurring kernels.
  arXiv  Detail & Related papers  (2021-11-15T15:23:55Z)
• AIM 2020 Challenge on Rendering Realistic Bokeh [95.87775182820518]
  This paper reviews the second AIM realistic bokeh effect rendering challenge. The goal was to learn a realistic shallow focus technique using a large-scale EBB! bokeh dataset. The participants had to render bokeh effect based on only one single frame without any additional data from other cameras or sensors.
  arXiv  Detail & Related papers  (2020-11-10T09:15:38Z)
• BGGAN: Bokeh-Glass Generative Adversarial Network for Rendering Realistic Bokeh [19.752904494597328]
  We propose a novel generator called Glass-Net, which generates bokeh images not relying on complex hardware. Experiments show that our method is able to render a high-quality bokeh effect and process one $1024 times 1536$ pixel image in 1.9 seconds on all smartphone chipsets.
  arXiv  Detail & Related papers  (2020-11-04T11:56:34Z)
• Rendering Natural Camera Bokeh Effect with Deep Learning [95.86933125733673]
  Bokeh is an important artistic effect used to highlight the main object of interest on the photo. Mobile cameras are unable to produce shallow depth-of-field photos due to a very small aperture diameter of their optics. We propose to learn a realistic shallow focus technique directly from the photos produced by DSLR cameras.
  arXiv  Detail & Related papers  (2020-06-10T07:28:06Z)
• Depth-aware Blending of Smoothed Images for Bokeh Effect Generation [10.790210744021072]
  In this paper, an end-to-end deep learning framework is proposed to generate high-quality bokeh effect from images. The network is lightweight and can process an HD image in 0.03 seconds. This approach ranked second in AIM 2019 Bokeh effect challenge-Perceptual Track.
  arXiv  Detail & Related papers  (2020-05-28T18:11:05Z)
• Deblurring by Realistic Blurring [110.54173799114785]
  We propose a new method which combines two GAN models, i.e., a learning-to-blurr GAN (BGAN) and learning-to-DeBlur GAN (DBGAN) The first model, BGAN, learns how to blur sharp images with unpaired sharp and blurry image sets, and then guides the second model, DBGAN, to learn how to correctly deblur such images. As an additional contribution, this paper also introduces a Real-World Blurred Image (RWBI) dataset including diverse blurry images.
  arXiv  Detail & Related papers  (2020-04-04T05:25:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.