Neural Video Portrait Relighting in Real-time via Consistency Modeling
- URL: http://arxiv.org/abs/2104.00484v1
- Date: Thu, 1 Apr 2021 14:13:28 GMT
- Title: Neural Video Portrait Relighting in Real-time via Consistency Modeling
- Authors: Longwen Zhang, Qixuan Zhang, Minye Wu, Jingyi Yu, Lan Xu
- Abstract summary: We propose a neural approach for real-time, high-quality and coherent video portrait relighting.
We propose a hybrid structure and lighting disentanglement in an encoder-decoder architecture.
We also propose a lighting sampling strategy to model the illumination consistency and mutation for natural portrait light manipulation in real-world.
- Score: 41.04622998356025
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Video portraits relighting is critical in user-facing human photography,
especially for immersive VR/AR experience. Recent advances still fail to
recover consistent relit result under dynamic illuminations from monocular RGB
stream, suffering from the lack of video consistency supervision. In this
paper, we propose a neural approach for real-time, high-quality and coherent
video portrait relighting, which jointly models the semantic, temporal and
lighting consistency using a new dynamic OLAT dataset. We propose a hybrid
structure and lighting disentanglement in an encoder-decoder architecture,
which combines a multi-task and adversarial training strategy for
semantic-aware consistency modeling. We adopt a temporal modeling scheme via
flow-based supervision to encode the conjugated temporal consistency in a cross
manner. We also propose a lighting sampling strategy to model the illumination
consistency and mutation for natural portrait light manipulation in real-world.
Extensive experiments demonstrate the effectiveness of our approach for
consistent video portrait light-editing and relighting, even using mobile
computing.
Related papers
- Low-Light Video Enhancement via Spatial-Temporal Consistent Illumination and Reflection Decomposition [68.6707284662443]
Low-Light Video Enhancement (LLVE) seeks to restore dynamic and static scenes plagued by severe invisibility and noise.
One critical aspect is formulating a consistency constraint specifically for temporal-spatial illumination and appearance enhanced versions.
We present an innovative video Retinex-based decomposition strategy that operates without the need for explicit supervision.
arXiv Detail & Related papers (2024-05-24T15:56:40Z) - URHand: Universal Relightable Hands [64.25893653236912]
We present URHand, the first universal relightable hand model that generalizes across viewpoints, poses, illuminations, and identities.
Our model allows few-shot personalization using images captured with a mobile phone, and is ready to be photorealistically rendered under novel illuminations.
arXiv Detail & Related papers (2024-01-10T18:59:51Z) - Relightful Harmonization: Lighting-aware Portrait Background Replacement [23.19641174787912]
We introduce Relightful Harmonization, a lighting-aware diffusion model designed to seamlessly harmonize sophisticated lighting effect for the foreground portrait using any background image.
Our approach unfolds in three stages. First, we introduce a lighting representation module that allows our diffusion model to encode lighting information from target image background.
Second, we introduce an alignment network that aligns lighting features learned from image background with lighting features learned from panorama environment maps.
arXiv Detail & Related papers (2023-12-11T23:20:31Z) - Personalized Video Relighting With an At-Home Light Stage [0.0]
We develop a personalized video relighting algorithm that produces high-quality and temporally consistent relit videos in real-time.
We show that by just capturing video of a user watching YouTube videos on a monitor we can train a personalized algorithm capable of performing high-quality relighting under any condition.
arXiv Detail & Related papers (2023-11-15T10:33:20Z) - Spatiotemporally Consistent HDR Indoor Lighting Estimation [66.26786775252592]
We propose a physically-motivated deep learning framework to solve the indoor lighting estimation problem.
Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position.
Our framework achieves photorealistic lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods.
arXiv Detail & Related papers (2023-05-07T20:36:29Z) - Joint Video Multi-Frame Interpolation and Deblurring under Unknown
Exposure Time [101.91824315554682]
In this work, we aim ambitiously for a more realistic and challenging task - joint video multi-frame and deblurring under unknown exposure time.
We first adopt a variant of supervised contrastive learning to construct an exposure-aware representation from input blurred frames.
We then build our video reconstruction network upon the exposure and motion representation by progressive exposure-adaptive convolution and motion refinement.
arXiv Detail & Related papers (2023-03-27T09:43:42Z) - Learning to Relight Portrait Images via a Virtual Light Stage and
Synthetic-to-Real Adaptation [76.96499178502759]
Relighting aims to re-illuminate the person in the image as if the person appeared in an environment with the target lighting.
Recent methods rely on deep learning to achieve high-quality results.
We propose a new approach that can perform on par with the state-of-the-art (SOTA) relighting methods without requiring a light stage.
arXiv Detail & Related papers (2022-09-21T17:15:58Z) - Spatially and color consistent environment lighting estimation using
deep neural networks for mixed reality [1.1470070927586016]
This paper presents a CNN-based model to estimate complex lighting for mixed reality environments.
We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting.
We show in experiments that the CNN architecture can predict the environment lighting with an average mean squared error (MSE) of num7.85e-04 when comparing SH lighting coefficients.
arXiv Detail & Related papers (2021-08-17T23:03:55Z) - Learning Illumination from Diverse Portraits [8.90355885907736]
We train our model using portrait photos paired with their ground truth environmental illumination.
We generate a rich set of such photos by using a light stage to record the reflectance field and alpha matte of 70 diverse subjects.
We show that our technique outperforms the state-of-the-art technique for portrait-based lighting estimation.
arXiv Detail & Related papers (2020-08-05T23:41:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.