Light-A-Video: Training-free Video Relighting via Progressive Light Fusion

• URL: http://arxiv.org/abs/2502.08590v1
• Date: Wed, 12 Feb 2025 17:24:19 GMT
• Title: Light-A-Video: Training-free Video Relighting via Progressive Light Fusion
• Authors: Yujie Zhou, Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Qidong Huang, Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Anyi Rao, Jiaqi Wang, Li Niu,
• Abstract summary: Light-A-Video is a training-free approach to achieve temporally smooth video relighting. Adapted from image relighting models, Light-A-Video introduces two key techniques to enhance lighting consistency.
• Score: 52.420894727186216
• License:
• Abstract: Recent advancements in image relighting models, driven by large-scale datasets and pre-trained diffusion models, have enabled the imposition of consistent lighting. However, video relighting still lags, primarily due to the excessive training costs and the scarcity of diverse, high-quality video relighting datasets. A simple application of image relighting models on a frame-by-frame basis leads to several issues: lighting source inconsistency and relighted appearance inconsistency, resulting in flickers in the generated videos. In this work, we propose Light-A-Video, a training-free approach to achieve temporally smooth video relighting. Adapted from image relighting models, Light-A-Video introduces two key techniques to enhance lighting consistency. First, we design a Consistent Light Attention (CLA) module, which enhances cross-frame interactions within the self-attention layers to stabilize the generation of the background lighting source. Second, leveraging the physical principle of light transport independence, we apply linear blending between the source video's appearance and the relighted appearance, using a Progressive Light Fusion (PLF) strategy to ensure smooth temporal transitions in illumination. Experiments show that Light-A-Video improves the temporal consistency of relighted video while maintaining the image quality, ensuring coherent lighting transitions across frames. Project page: https://bujiazi.github.io/light-a-video.github.io/.

Related papers

• Lumina-Video: Efficient and Flexible Video Generation with Multi-scale Next-DiT [98.56372305225271]
  Lumina-Next achieves exceptional performance in the generation of images with Next-DiT. Lumina-Video incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple patchifications. We propose Lumina-V2A, a video-to-audio model based on Next-DiT, to create synchronized sounds for generated videos.
  arXiv  Detail & Related papers  (2025-02-10T18:58:11Z)
• RelightVid: Temporal-Consistent Diffusion Model for Video Relighting [95.10341081549129]
  RelightVid is a flexible framework for video relighting. It can accept background video, text prompts, or environment maps as relighting conditions. It achieves arbitrary video relighting with high temporal consistency without intrinsic decomposition.
  arXiv  Detail & Related papers  (2025-01-27T18:59:57Z)
• LumiSculpt: A Consistency Lighting Control Network for Video Generation [67.48791242688493]
  Lighting plays a pivotal role in ensuring the naturalness of video generation. It remains challenging to disentangle and model independent and coherent lighting attributes. LumiSculpt enables precise and consistent lighting control in T2V generation models.
  arXiv  Detail & Related papers  (2024-10-30T12:44:08Z)
• Real-time 3D-aware Portrait Video Relighting [89.41078798641732]
  We present the first real-time 3D-aware method for relighting in-the-wild videos of talking faces based on Neural Radiance Fields (NeRF) We infer an albedo tri-plane, as well as a shading tri-plane based on a desired lighting condition for each video frame with fast dual-encoders. Our method runs at 32.98 fps on consumer-level hardware and achieves state-of-the-art results in terms of reconstruction quality, lighting error, lighting instability, temporal consistency and inference speed.
  arXiv  Detail & Related papers  (2024-10-24T01:34:11Z)
• Personalized Video Relighting With an At-Home Light Stage [0.0]
  We develop a personalized video relighting algorithm that produces high-quality and temporally consistent relit videos in real-time. We show that by just capturing recordings of a user watching YouTube videos on a monitor we can train a personalized algorithm capable of performing high-quality relighting under any condition.
  arXiv  Detail & Related papers  (2023-11-15T10:33:20Z)
• Low-Light Video Enhancement with Synthetic Event Guidance [188.7256236851872]
  We use synthetic events from multiple frames to guide the enhancement and restoration of low-light videos. Our method outperforms existing low-light video or single image enhancement approaches on both synthetic and real LLVE datasets.
  arXiv  Detail & Related papers  (2022-08-23T14:58:29Z)
• Neural Video Portrait Relighting in Real-time via Consistency Modeling [41.04622998356025]
  We propose a neural approach for real-time, high-quality and coherent video portrait relighting. We propose a hybrid structure and lighting disentanglement in an encoder-decoder architecture. We also propose a lighting sampling strategy to model the illumination consistency and mutation for natural portrait light manipulation in real-world.
  arXiv  Detail & Related papers  (2021-04-01T14:13:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.