LiveLight: Real-time Streaming Video Relighting with Interactive Control

• URL: http://arxiv.org/abs/2608.01771v1
• Date: Mon, 03 Aug 2026 06:45:00 GMT
• Title: LiveLight: Real-time Streaming Video Relighting with Interactive Control
• Abstract summary: We present LiveLight, the first diffusion-based framework for real-time streaming video relighting with interactive 3D lighting control.<n>We overcome three critical challenges: effectively injecting dynamic 3D lighting into a diffusion model, maintaining high-fidelity generation under a low NFE budget, and facilitating continuous streaming for interactive control.<n>Experiments on real-world and synthetic benchmarks demonstrate that LiveLight achieves state-of-the-art relighting quality while running at real-time speed.
• Score: 50.08584152649119
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present LiveLight, the first diffusion-based framework for real-time streaming video relighting with interactive 3D lighting control. Achieving this is non-trivial, as it requires overcoming three critical challenges: effectively injecting dynamic 3D lighting into a diffusion model, maintaining high-fidelity generation under an extremely low NFE (Number of Function Evaluations) budget for real-time speed, and facilitating continuous streaming for interactive control. To address these pain points, we propose three key designs. First, for accurate lighting injection, we propose a lightweight adapter that feeds Multi-Plane Light Irradiance (MPLI) conditions-depth-aware irradiance maps encoding 3D lighting geometry-directly into the diffusion backbone. Second, to prevent rendering quality degradation at low NFEs towards real-time distillation, we introduce a geometry-guided feedback branch. This training-time constraint leverages a frozen geometry estimator to enforce depth- and normal-consistent relighting, ensuring geometrically plausible shading without adding inference overhead. Finally, to enable streaming interaction, we develop a progressive rolling-window strategy that maintains a denoising ladder of latent chunks at varying noise levels. By propagating intermediate states, this strategy guarantees temporal coherence and supports arbitrarily long video relighting with per-frame reference refresh. Extensive experiments on real-world and synthetic benchmarks demonstrate that LiveLight achieves state-of-the-art relighting quality while running at real-time speed, significantly outperforming offline baselines in temporal stability, lighting controllability, and user preference. To foster real-time interactive relighting research, we will publicly release our models, training data, and synthetic data generator.

Related papers

• SpotlessGS: Relightable 3D Gaussian Splatting under Dynamic Illumination for Robotic Perception [20.515607443256073]
  We extend Dark Gaussian Splatting to a more accurate and flexible relightable 3D reconstruction framework.<n>We introduce a low-frequency illumination model based on spherical harmonics to capture spatially varying residual and ambient lighting effects.<n> Experiments on synthetic and real-world datasets demonstrate that our method effectively mitigates illumination artifacts while improving rendering quality and quantitative performance over prior approaches.
  arXiv  Detail & Related papers  (2026-08-11T17:35:34Z)
• DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance [4.219150964619931]
  DeepIPCv3 is a novel multi-modal autonomous navigation framework.<n>It synergizes the dense 3D spatial geometry of LiDAR point clouds with the microsecond-level asynchronous event streams of a Dynamic Vision Sensor.<n>The proposed LiDAR and DVS fusion yields the lowest trajectory and control command errors, enabling highly reactive, mathematically bounded evasive maneuvers.
  arXiv  Detail & Related papers  (2026-05-31T14:59:01Z)
• Adaptive Illumination Control for Robot Perception [6.718749267591075]
  Lightning is a closed-loop illumination-control framework for visual SLAM.<n>It combines relighting, offline optimization, and imitation learning.<n>It substantially improves SLAM trajectory while reducing unnecessary illumination power.
  arXiv  Detail & Related papers  (2026-02-13T03:34:01Z)
• RRNet: Configurable Real-Time Video Enhancement with Arbitrary Local Lighting Variations [7.360594425594612]
  We introduce RRNet, a framework that achieves a state-of-the-art tradeoff between visual quality and efficiency.<n> RRNet enables localized relighting through a depth-aware rendering module without requiring pixel-aligned training data.<n>Experiments show that RRNet consistently outperforms prior methods in low-light enhancement, localized illumination adjustment, and glare removal.
  arXiv  Detail & Related papers  (2026-01-05T07:50:59Z)
• Rolling Forcing: Autoregressive Long Video Diffusion in Real Time [86.40480237741609]
  Rolling Forcing is a novel video generation technique that enables streaming long videos with minimal error accumulation.<n>Rolling Forcing comes with three novel designs. First, instead of iteratively sampling individual frames, which accelerates error propagation, we design a joint denoising scheme.<n>Second, we introduce the attention sink mechanism into the long-horizon stream video generation task, which allows the model to keep key value states of initial frames as a global context anchor.<n>Third, we design an efficient training algorithm that enables few-step distillation over largely extended denoising windows.
  arXiv  Detail & Related papers  (2025-09-29T17:57:14Z)
• Zero-TIG: Temporal Consistency-Aware Zero-Shot Illumination-Guided Low-light Video Enhancement [2.9695823613761316]
  Low-light and underwater videos suffer from poor visibility, low contrast, and high noise.<n>Existing approaches typically rely on paired ground truth, which limits their practicality and often fails to maintain temporal consistency.<n>This paper introduces a novel zero-shot learning approach named Zero-TIG, leveraging the Retinex theory and optical flow techniques.
  arXiv  Detail & Related papers  (2025-03-14T08:22:26Z)
• Real-time 3D-aware Portrait Video Relighting [89.41078798641732]
  We present the first real-time 3D-aware method for relighting in-the-wild videos of talking faces based on Neural Radiance Fields (NeRF) We infer an albedo tri-plane, as well as a shading tri-plane based on a desired lighting condition for each video frame with fast dual-encoders. Our method runs at 32.98 fps on consumer-level hardware and achieves state-of-the-art results in terms of reconstruction quality, lighting error, lighting instability, temporal consistency and inference speed.
  arXiv  Detail & Related papers  (2024-10-24T01:34:11Z)
• Unrolled Decomposed Unpaired Learning for Controllable Low-Light Video Enhancement [48.76608212565327]
  This paper makes endeavors in the direction of learning for low-light video enhancement without using paired ground truth. Compared to low-light image enhancement, enhancing low-light videos is more difficult due to the intertwined effects of noise, exposure, and contrast in the spatial domain, jointly with the need for temporal coherence. We propose the Unrolled Decomposed Unpaired Network (UDU-Net) for enhancing low-light videos by unrolling the optimization functions into a deep network to decompose the signal into spatial and temporal-related factors, which are updated iteratively.
  arXiv  Detail & Related papers  (2024-08-22T11:45:11Z)
• Low-Light Video Enhancement via Spatial-Temporal Consistent Decomposition [52.89441679581216]
  Low-Light Video Enhancement (LLVE) seeks to restore dynamic or static scenes plagued by severe invisibility and noise.<n>We present an innovative video decomposition strategy that incorporates view-independent and view-dependent components.<n>Our framework consistently outperforms existing methods, establishing a new SOTA performance.
  arXiv  Detail & Related papers  (2024-05-24T15:56:40Z)
• Spatiotemporally Consistent HDR Indoor Lighting Estimation [66.26786775252592]
  We propose a physically-motivated deep learning framework to solve the indoor lighting estimation problem. Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position. Our framework achieves photorealistic lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods.
  arXiv  Detail & Related papers  (2023-05-07T20:36:29Z)
• Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution [96.89588203312451]
  This paper presents a new parametric representation called Deep Parametric 3D Filters (DP3DF) DP3DF incorporates local information to enable simultaneous denoising, illumination enhancement, and SR efficiently in a single encoder-and-decoder network. Also, a dynamic residual frame is jointly learned with the DP3DF via a shared backbone to further boost the SR quality.
  arXiv  Detail & Related papers  (2022-07-05T03:57:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.