LightPainter: Interactive Portrait Relighting with Freehand Scribble

• URL: http://arxiv.org/abs/2303.12950v1
• Date: Wed, 22 Mar 2023 23:17:11 GMT
• Title: LightPainter: Interactive Portrait Relighting with Freehand Scribble
• Authors: Yiqun Mei, He Zhang, Xuaner Zhang, Jianming Zhang, Zhixin Shu, Yilin Wang, Zijun Wei, Shi Yan, HyunJoon Jung, Vishal M. Patel
• Abstract summary: We introduce LightPainter, a scribble-based relighting system that allows users to interactively manipulate portrait lighting effect with ease. To train the relighting module, we propose a novel scribble simulation procedure to mimic real user scribbles. We demonstrate high-quality and flexible portrait lighting editing capability with both quantitative and qualitative experiments.
• Score: 79.95574780974103
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent portrait relighting methods have achieved realistic results of portrait lighting effects given a desired lighting representation such as an environment map. However, these methods are not intuitive for user interaction and lack precise lighting control. We introduce LightPainter, a scribble-based relighting system that allows users to interactively manipulate portrait lighting effect with ease. This is achieved by two conditional neural networks, a delighting module that recovers geometry and albedo optionally conditioned on skin tone, and a scribble-based module for relighting. To train the relighting module, we propose a novel scribble simulation procedure to mimic real user scribbles, which allows our pipeline to be trained without any human annotations. We demonstrate high-quality and flexible portrait lighting editing capability with both quantitative and qualitative experiments. User study comparisons with commercial lighting editing tools also demonstrate consistent user preference for our method.

Related papers

• Relightable Neural Actor with Intrinsic Decomposition and Pose Control [80.06094206522668]
  We propose Relightable Neural Actor, a new video-based method for learning a pose-driven neural human model that can be relighted. For training, our method solely requires a multi-view recording of the human under a known, but static lighting condition. To evaluate our approach in real-world scenarios, we collect a new dataset with four identities recorded under different light conditions, indoors and outdoors.
  arXiv  Detail & Related papers  (2023-12-18T14:30:13Z)
• EverLight: Indoor-Outdoor Editable HDR Lighting Estimation [9.443561684223514]
  We propose a method which combines a parametric light model with 360deg panoramas, ready to use as HDRI in rendering engines. In our representation, users can easily edit light direction, intensity, number, etc. to impact shading while providing rich, complex reflections while seamlessly blending with the edits.
  arXiv  Detail & Related papers  (2023-04-26T00:20:59Z)
• WildLight: In-the-wild Inverse Rendering with a Flashlight [77.31815397135381]
  We propose a practical photometric solution for in-the-wild inverse rendering under unknown ambient lighting. Our system recovers scene geometry and reflectance using only multi-view images captured by a smartphone. We demonstrate by extensive experiments that our method is easy to implement, casual to set up, and consistently outperforms existing in-the-wild inverse rendering techniques.
  arXiv  Detail & Related papers  (2023-03-24T17:59:56Z)
• RelightableHands: Efficient Neural Relighting of Articulated Hand Models [46.60594572471557]
  We present the first neural relighting approach for rendering high-fidelity personalized hands that can be animated in real-time under novel illumination. Our approach adopts a teacher-student framework, where the teacher learns appearance under a single point light from images captured in a light-stage. Using images rendered by the teacher model as training data, an efficient student model directly predicts appearance under natural illuminations in real-time.
  arXiv  Detail & Related papers  (2023-02-09T18:59:48Z)
• Editable Indoor Lighting Estimation [6.531546527140474]
  We propose a pipeline that estimates a parametric light that is easy to edit and allows renderings with strong shadows. We show that our approach makes indoor lighting estimation easier to handle by a casual user, while still producing competitive results.
  arXiv  Detail & Related papers  (2022-11-08T00:58:29Z)
• Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation [76.96499178502759]
  Relighting aims to re-illuminate the person in the image as if the person appeared in an environment with the target lighting. Recent methods rely on deep learning to achieve high-quality results. We propose a new approach that can perform on par with the state-of-the-art (SOTA) relighting methods without requiring a light stage.
  arXiv  Detail & Related papers  (2022-09-21T17:15:58Z)
• Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination [63.992213016011235]
  We propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function. Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition. Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art.
  arXiv  Detail & Related papers  (2022-07-27T16:07:48Z)
• Hierarchical Vectorization for Portrait Images [12.32304366243904]
  We propose a novel vectorization method that can automatically convert images into a 3-tier hierarchical representation. The base layer consists of a set of sparse diffusion curves which characterize salient geometric features and low-frequency colors. The middle level encodes specular highlights and shadows to large and editable Poisson regions (PR) and allows the user to directly adjust illumination. The top level contains two types of pixel-sized PRs for high-frequency residuals and fine details such as pimples and pigmentation.
  arXiv  Detail & Related papers  (2022-05-24T07:58:41Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.