Relit-LiVE: Relight Video by Jointly Learning Environment Video

• URL: http://arxiv.org/abs/2605.06658v1
• Date: Thu, 07 May 2026 17:58:15 GMT
• Title: Relit-LiVE: Relight Video by Jointly Learning Environment Video
• Abstract summary: We present Relit-LiVE, a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose.<n>Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations.<n>Our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting.
• Score: 17.623352754423287
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances have shown that large-scale video diffusion models can be repurposed as neural renderers by first decomposing videos into intrinsic scene representations and then performing forward rendering under novel illumination. While promising, this paradigm fundamentally relies on accurate intrinsic decomposition, which remains highly unreliable for real-world videos and often leads to distorted appearances, broken materials, and accumulated temporal artifacts during relighting. In this work, we present Relit-LiVE, a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations. Furthermore, we propose a novel environment video prediction formulation that simultaneously generates relit videos and per-frame environment maps aligned with each camera viewpoint in a single diffusion process. This joint prediction enforces strong geometric-illumination alignment and naturally supports dynamic lighting and camera motion, significantly improving physical consistency in video relighting while easing the requirement of known per-frame camera pose. Extensive experiments demonstrate that Relit-LiVE consistently outperforms state-of-the-art video relighting and neural rendering methods across synthetic and real-world benchmarks. Beyond relighting, our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting. The Project is available at https://github.com/zhuxing0/Relit-LiVE.

Related papers

• LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting [40.05860097816268]
  LightCrafter is a hybrid pipeline that reforms video relighting as video translation of a proxy video.<n>Rather than translating the input video directly to the target, we translate a rendering of the input under the target illumination to the final target.<n>We show PBR renders alone already outperform some prior art but struggle with effects like global illumination.
  arXiv  Detail & Related papers  (2026-07-09T00:44:43Z)
• Video Generation Models Are Inherent Lighting Estimators [54.44164570819254]
  Recent generation models can produce scenes with complex lighting, possessing an inherent understanding of lighting.<n>We introduce V-LITE (Video generation models are inherent lighting estimators), a framework that unlocks this internal knowledge by reframing estimation.<n>To bridge the gap from LDR-native models to the HDR domain, we design an HDR-aware VAE and employ an efficient LoRA-based fine-tuning strategy.
  arXiv  Detail & Related papers  (2026-07-06T04:59:17Z)
• PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion [51.8637101384136]
  We present PAI-Studio, a new reference-conditioned video synthesis task.<n> generating dynamic backgrounds aligned with foreground motion while preserving foreground identity.<n>We further construct a 30K-scale dataset sourced from high-quality films and online videos to support this task.
  arXiv  Detail & Related papers  (2026-05-31T18:45:11Z)
• HarmoVid: Relightful Video Portrait Harmonization [32.950000237388686]
  We present a method for harmonizing the lighting of a foreground video to match a target background scene.<n>Unlike images, acquiring labeled data for videos is practically infeasible and non-scalable.<n>We introduce a novel lighting deflickering model that can stabilize the global and local lighting flickering artifacts.
  arXiv  Detail & Related papers  (2026-05-27T17:59:02Z)
• Lighting-grounded Video Generation with Renderer-based Agent Reasoning [56.50946217758078]
  LiVER is a diffusion-based framework for scene-controllable video generation.<n>We present a novel framework that conditions video synthesis on explicit 3D scene properties.<n>Our method disentangles these properties by rendering control signals from a unified 3D representation.
  arXiv  Detail & Related papers  (2026-04-09T08:29:29Z)
• ReLumix: Extending Image Relighting to Video via Video Diffusion Models [5.890782804843724]
  Controlling illumination during video post-production is a crucial yet elusive goal in computational photography.<n>This paper introduces ReLumix, a novel framework that decouples the relighting from temporal synthesis.<n>Although trained on synthetic data, ReLumix shows competitive generalization to real-world videos.
  arXiv  Detail & Related papers  (2025-09-28T09:35:33Z)
• Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models [18.008901495139717]
  We propose Lumen, an end-to-end video relighting framework developed on large-scale video generative models.<n>For the synthetic domain, we leverage advanced 3D rendering engine to curate video pairs in diverse environments.<n>For the realistic domain, we adapt a HDR-based lighting simulation to complement the lack of paired in-the-wild videos.
  arXiv  Detail & Related papers  (2025-08-18T14:21:22Z)
• Light-A-Video: Training-free Video Relighting via Progressive Light Fusion [52.420894727186216]
  Light-A-Video is a training-free approach to achieve temporally smooth video relighting.<n>Adapted from image relighting models, Light-A-Video introduces two key techniques to enhance lighting consistency.
  arXiv  Detail & Related papers  (2025-02-12T17:24:19Z)
• DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models [83.28670336340608]
  We introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering.<n>Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.
  arXiv  Detail & Related papers  (2025-01-30T18:59:11Z)
• RelightVid: Temporal-Consistent Diffusion Model for Video Relighting [95.10341081549129]
  RelightVid is a flexible framework for video relighting.<n>It can accept background video, text prompts, or environment maps as relighting conditions.<n>It achieves arbitrary video relighting with high temporal consistency without intrinsic decomposition.
  arXiv  Detail & Related papers  (2025-01-27T18:59:57Z)
• MoBluRF: Motion Deblurring Neural Radiance Fields for Blurry Monocular Video [25.964642223641057]
  MoBluRF is a framework for synthesis of sharp-temporal views in blurry monocular video.<n>In the BRI stage, we reconstruct dynamic 3D scenes and jointly initialize the base rays which are used to predict latent sharp rays.<n>In the MDD stage, we introduce a novel Incremental Latent Sharp-rays Prediction (ILSP) approach for the blurry monocular video frames.
  arXiv  Detail & Related papers  (2023-12-21T02:01:19Z)
• Personalized Video Relighting With an At-Home Light Stage [0.0]
  We develop a personalized video relighting algorithm that produces high-quality and temporally consistent relit videos in real-time. We show that by just capturing recordings of a user watching YouTube videos on a monitor we can train a personalized algorithm capable of performing high-quality relighting under any condition.
  arXiv  Detail & Related papers  (2023-11-15T10:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.