Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction

• URL: http://arxiv.org/abs/2606.02919v3
• Date: Fri, 05 Jun 2026 00:48:04 GMT
• Title: Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction
• Abstract summary: We present a diffusion-based method for relighting dynamic portrait videos with photorealism and temporal consistency.<n>Our model can produce temporally consistent relit portrait video that looks realistic and under a provided new environment.<n>Results show that our method achieves state-of-the-art performance in photorealism, lighting harmony, and temporal consistency.
• Score: 23.572176892343254
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a diffusion-based method for relighting dynamic portrait videos with photorealism and temporal consistency. Our method is fueled by a hybrid training dataset that consists of real-captured and rendered dynamic portrait videos with diverse subject appearances, facial motions, head poses, and known lighting conditions. Specifically, we construct an LED-based lighting system for realistic lighting emulation and high-speed video relighting data acquisition. By leveraging the image priors embedded in pre-trained video diffusion models, and using per-frame high dynamic range (HDR) environment map as lighting control, we train a high-performance generative model for realistic and identity-preserving dynamic portrait video relighting. In addition to the environment map control, our model uses a synthesized background image to enable control on the camera's exposure level and color tone. Our model can produce temporally consistent relit portrait video that looks realistic and harmonious under a provided new environment and faithfully preserve the subject's expression and fine facial features, including skin tone, wrinkles, and facial hair. Our model generalizes well to unseen data, in terms of the subject appearance, motion, and lighting condition. We perform extensive experiments on relighting in-the-wild videos with various environment maps and demonstrate practical applications on portrait photography. Results show that our method achieves state-of-the-art performance in photorealism, lighting harmony, and temporal consistency.

Related papers

• Video Generation Models Are Inherent Lighting Estimators [54.44164570819254]
  Recent generation models can produce scenes with complex lighting, possessing an inherent understanding of lighting.<n>We introduce V-LITE (Video generation models are inherent lighting estimators), a framework that unlocks this internal knowledge by reframing estimation.<n>To bridge the gap from LDR-native models to the HDR domain, we design an HDR-aware VAE and employ an efficient LoRA-based fine-tuning strategy.
  arXiv  Detail & Related papers  (2026-07-06T04:59:17Z)
• HarmoVid: Relightful Video Portrait Harmonization [32.950000237388686]
  We present a method for harmonizing the lighting of a foreground video to match a target background scene.<n>Unlike images, acquiring labeled data for videos is practically infeasible and non-scalable.<n>We introduce a novel lighting deflickering model that can stabilize the global and local lighting flickering artifacts.
  arXiv  Detail & Related papers  (2026-05-27T17:59:02Z)
• POLAR: A Portrait OLAT Dataset and Generative Framework for Illumination-Aware Face Modeling [51.7495375918484]
  Face relighting aims to synthesize realistic portraits under novel illumination while preserving identity and geometry.<n>We introduce POLAR, a large-scale and physically calibrated One-Light-at-a-Time dataset containing over 200 subjects captured under 156 lighting directions, multiple views, and diverse expressions.<n>We develop a flow-based generative model POLARNet that predicts per-light OLAT responses from a single portrait, capturing fine-grained and direction-aware illumination effects while preserving facial identity.
  arXiv  Detail & Related papers  (2025-12-15T11:04:09Z)
• LuxDiT: Lighting Estimation with Video Diffusion Transformer [66.60450792095901]
  Estimating scene lighting from a single image or video remains a longstanding challenge in computer vision and graphics.<n>We propose LuxDiT, a novel data-driven approach that fine-tunes a video diffusion transformer to generate HDR environment maps conditioned on visual input.
  arXiv  Detail & Related papers  (2025-09-03T19:59:20Z)
• Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset [33.388553876200795]
  We introduce Lux Post Facto, a novel portrait video relighting method that produces both photorealistic and temporally consistent lighting effects.<n>Our technique uses a hybrid dataset consisting of static expression OLAT data and in-the-wild portrait performance videos to jointly learn relighting and temporal modeling.
  arXiv  Detail & Related papers  (2025-03-18T17:55:22Z)
• RelightVid: Temporal-Consistent Diffusion Model for Video Relighting [95.10341081549129]
  RelightVid is a flexible framework for video relighting.<n>It can accept background video, text prompts, or environment maps as relighting conditions.<n>It achieves arbitrary video relighting with high temporal consistency without intrinsic decomposition.
  arXiv  Detail & Related papers  (2025-01-27T18:59:57Z)
• Real-time 3D-aware Portrait Video Relighting [89.41078798641732]
  We present the first real-time 3D-aware method for relighting in-the-wild videos of talking faces based on Neural Radiance Fields (NeRF) We infer an albedo tri-plane, as well as a shading tri-plane based on a desired lighting condition for each video frame with fast dual-encoders. Our method runs at 32.98 fps on consumer-level hardware and achieves state-of-the-art results in terms of reconstruction quality, lighting error, lighting instability, temporal consistency and inference speed.
  arXiv  Detail & Related papers  (2024-10-24T01:34:11Z)
• DifFRelight: Diffusion-Based Facial Performance Relighting [12.909429637057343]
  We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. We train a diffusion model for precise lighting control, enabling high-fidelity relit facial images from flat-lit inputs. The model accurately reproduces complex lighting effects like eye reflections, subsurface scattering, self-shadowing, and translucency.
  arXiv  Detail & Related papers  (2024-10-10T17:56:44Z)
• Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering [56.68286440268329]
  correct insertion of virtual objects in images of real-world scenes requires a deep understanding of the scene's lighting, geometry and materials. We propose using a personalized large diffusion model as guidance to a physically based inverse rendering process. Our method recovers scene lighting and tone-mapping parameters, allowing the photorealistic composition of arbitrary virtual objects in single frames or videos of indoor or outdoor scenes.
  arXiv  Detail & Related papers  (2024-08-19T05:15:45Z)
• Neural Video Portrait Relighting in Real-time via Consistency Modeling [41.04622998356025]
  We propose a neural approach for real-time, high-quality and coherent video portrait relighting. We propose a hybrid structure and lighting disentanglement in an encoder-decoder architecture. We also propose a lighting sampling strategy to model the illumination consistency and mutation for natural portrait light manipulation in real-world.
  arXiv  Detail & Related papers  (2021-04-01T14:13:28Z)
• Learning Illumination from Diverse Portraits [8.90355885907736]
  We train our model using portrait photos paired with their ground truth environmental illumination. We generate a rich set of such photos by using a light stage to record the reflectance field and alpha matte of 70 diverse subjects. We show that our technique outperforms the state-of-the-art technique for portrait-based lighting estimation.
  arXiv  Detail & Related papers  (2020-08-05T23:41:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.