Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset

• URL: http://arxiv.org/abs/2503.14485v2
• Date: Wed, 02 Apr 2025 02:46:45 GMT
• Title: Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset
• Abstract summary: We introduce Lux Post Facto, a novel portrait video relighting method that produces both photorealistic and temporally consistent lighting effects.<n>Our technique uses a hybrid dataset consisting of static expression OLAT data and in-the-wild portrait performance videos to jointly learn relighting and temporal modeling.
• Score: 33.388553876200795
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video portrait relighting remains challenging because the results need to be both photorealistic and temporally stable. This typically requires a strong model design that can capture complex facial reflections as well as intensive training on a high-quality paired video dataset, such as dynamic one-light-at-a-time (OLAT). In this work, we introduce Lux Post Facto, a novel portrait video relighting method that produces both photorealistic and temporally consistent lighting effects. From the model side, we design a new conditional video diffusion model built upon state-of-the-art pre-trained video diffusion model, alongside a new lighting injection mechanism to enable precise control. This way we leverage strong spatial and temporal generative capability to generate plausible solutions to the ill-posed relighting problem. Our technique uses a hybrid dataset consisting of static expression OLAT data and in-the-wild portrait performance videos to jointly learn relighting and temporal modeling. This avoids the need to acquire paired video data in different lighting conditions. Our extensive experiments show that our model produces state-of-the-art results both in terms of photorealism and temporal consistency.

Related papers

• Video Generation Models Are Inherent Lighting Estimators [54.44164570819254]
  Recent generation models can produce scenes with complex lighting, possessing an inherent understanding of lighting.<n>We introduce V-LITE (Video generation models are inherent lighting estimators), a framework that unlocks this internal knowledge by reframing estimation.<n>To bridge the gap from LDR-native models to the HDR domain, we design an HDR-aware VAE and employ an efficient LoRA-based fine-tuning strategy.
  arXiv  Detail & Related papers  (2026-07-06T04:59:17Z)
• Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction [23.572176892343254]
  We present a diffusion-based method for relighting dynamic portrait videos with photorealism and temporal consistency.<n>Our model can produce temporally consistent relit portrait video that looks realistic and under a provided new environment.<n>Results show that our method achieves state-of-the-art performance in photorealism, lighting harmony, and temporal consistency.
  arXiv  Detail & Related papers  (2026-06-01T21:47:17Z)
• HarmoVid: Relightful Video Portrait Harmonization [32.950000237388686]
  We present a method for harmonizing the lighting of a foreground video to match a target background scene.<n>Unlike images, acquiring labeled data for videos is practically infeasible and non-scalable.<n>We introduce a novel lighting deflickering model that can stabilize the global and local lighting flickering artifacts.
  arXiv  Detail & Related papers  (2026-05-27T17:59:02Z)
• BodyReLux: Temporally Consistent Full-Body Video Relighting [8.409048029216258]
  We present BodyReLux, a framework for relighting full-body human performances in a temporally consistent way.<n>Our model is trained on a hybrid dataset of pixel-aligned video relighting pairs.<n>We achieve photorealistic, robust, and temporally consistent video relighting of subject-specific human performances.
  arXiv  Detail & Related papers  (2026-05-20T21:57:31Z)
• POLAR: A Portrait OLAT Dataset and Generative Framework for Illumination-Aware Face Modeling [51.7495375918484]
  Face relighting aims to synthesize realistic portraits under novel illumination while preserving identity and geometry.<n>We introduce POLAR, a large-scale and physically calibrated One-Light-at-a-Time dataset containing over 200 subjects captured under 156 lighting directions, multiple views, and diverse expressions.<n>We develop a flow-based generative model POLARNet that predicts per-light OLAT responses from a single portrait, capturing fine-grained and direction-aware illumination effects while preserving facial identity.
  arXiv  Detail & Related papers  (2025-12-15T11:04:09Z)
• ReLumix: Extending Image Relighting to Video via Video Diffusion Models [5.890782804843724]
  Controlling illumination during video post-production is a crucial yet elusive goal in computational photography.<n>This paper introduces ReLumix, a novel framework that decouples the relighting from temporal synthesis.<n>Although trained on synthetic data, ReLumix shows competitive generalization to real-world videos.
  arXiv  Detail & Related papers  (2025-09-28T09:35:33Z)
• UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting [85.27994475113056]
  We introduce a general-purpose approach that jointly estimates albedo and synthesizes relit outputs in a single pass.<n>Our model demonstrates strong generalization across diverse domains and surpasses previous methods in both visual fidelity and temporal consistency.
  arXiv  Detail & Related papers  (2025-06-18T17:56:45Z)
• Light-A-Video: Training-free Video Relighting via Progressive Light Fusion [52.420894727186216]
  Light-A-Video is a training-free approach to achieve temporally smooth video relighting.<n>Adapted from image relighting models, Light-A-Video introduces two key techniques to enhance lighting consistency.
  arXiv  Detail & Related papers  (2025-02-12T17:24:19Z)
• DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models [83.28670336340608]
  We introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.
  arXiv  Detail & Related papers  (2025-01-30T18:59:11Z)
• RelightVid: Temporal-Consistent Diffusion Model for Video Relighting [95.10341081549129]
  RelightVid is a flexible framework for video relighting.<n>It can accept background video, text prompts, or environment maps as relighting conditions.<n>It achieves arbitrary video relighting with high temporal consistency without intrinsic decomposition.
  arXiv  Detail & Related papers  (2025-01-27T18:59:57Z)
• LumiSculpt: A Consistency Lighting Control Network for Video Generation [67.48791242688493]
  Lighting plays a pivotal role in ensuring the naturalness of video generation. It remains challenging to disentangle and model independent and coherent lighting attributes. LumiSculpt enables precise and consistent lighting control in T2V generation models.
  arXiv  Detail & Related papers  (2024-10-30T12:44:08Z)
• Real-time 3D-aware Portrait Video Relighting [89.41078798641732]
  We present the first real-time 3D-aware method for relighting in-the-wild videos of talking faces based on Neural Radiance Fields (NeRF) We infer an albedo tri-plane, as well as a shading tri-plane based on a desired lighting condition for each video frame with fast dual-encoders. Our method runs at 32.98 fps on consumer-level hardware and achieves state-of-the-art results in terms of reconstruction quality, lighting error, lighting instability, temporal consistency and inference speed.
  arXiv  Detail & Related papers  (2024-10-24T01:34:11Z)
• Relightable Neural Actor with Intrinsic Decomposition and Pose Control [80.06094206522668]
  We propose Relightable Neural Actor, a new video-based method for learning a pose-driven neural human model that can be relighted. For training, our method solely requires a multi-view recording of the human under a known, but static lighting condition. To evaluate our approach in real-world scenarios, we collect a new dataset with four identities recorded under different light conditions, indoors and outdoors.
  arXiv  Detail & Related papers  (2023-12-18T14:30:13Z)
• Personalized Video Relighting With an At-Home Light Stage [0.0]
  We develop a personalized video relighting algorithm that produces high-quality and temporally consistent relit videos in real-time. We show that by just capturing recordings of a user watching YouTube videos on a monitor we can train a personalized algorithm capable of performing high-quality relighting under any condition.
  arXiv  Detail & Related papers  (2023-11-15T10:33:20Z)
• Neural Video Portrait Relighting in Real-time via Consistency Modeling [41.04622998356025]
  We propose a neural approach for real-time, high-quality and coherent video portrait relighting. We propose a hybrid structure and lighting disentanglement in an encoder-decoder architecture. We also propose a lighting sampling strategy to model the illumination consistency and mutation for natural portrait light manipulation in real-world.
  arXiv  Detail & Related papers  (2021-04-01T14:13:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.