Lighting-grounded Video Generation with Renderer-based Agent Reasoning

• URL: http://arxiv.org/abs/2604.07966v1
• Date: Thu, 09 Apr 2026 08:29:29 GMT
• Title: Lighting-grounded Video Generation with Renderer-based Agent Reasoning
• Abstract summary: LiVER is a diffusion-based framework for scene-controllable video generation.<n>We present a novel framework that conditions video synthesis on explicit 3D scene properties.<n>Our method disentangles these properties by rendering control signals from a unified 3D representation.
• Score: 56.50946217758078
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have achieved remarkable progress in video generation, but their controllability remains a major limitation. Key scene factors such as layout, lighting, and camera trajectory are often entangled or only weakly modeled, restricting their applicability in domains like filmmaking and virtual production where explicit scene control is essential. We present LiVER, a diffusion-based framework for scene-controllable video generation. To achieve this, we introduce a novel framework that conditions video synthesis on explicit 3D scene properties, supported by a new large-scale dataset with dense annotations of object layout, lighting, and camera parameters. Our method disentangles these properties by rendering control signals from a unified 3D representation. We propose a lightweight conditioning module and a progressive training strategy to integrate these signals into a foundational video diffusion model, ensuring stable convergence and high fidelity. Our framework enables a wide range of applications, including image-to-video and video-to-video synthesis where the underlying 3D scene is fully editable. To further enhance usability, we develop a scene agent that automatically translates high-level user instructions into the required 3D control signals. Experiments show that LiVER achieves state-of-the-art photorealism and temporal consistency while enabling precise, disentangled control over scene factors, setting a new standard for controllable video generation.

Related papers

• Control-DINO: Feature Space Conditioning for Controllable Image-to-Video Diffusion [4.4853338999399375]
  We introduce a lightweight architecture and training strategy that decouples appearance from other features.<n>We show that low spatial resolution can be compensated by higher feature dimensionality, improving controllability in generative rendering from explicit spatial representations.
  arXiv  Detail & Related papers  (2026-04-02T08:27:48Z)
• RealMaster: Lifting Rendered Scenes into Photorealistic Video [55.04231137698114]
  State-of-the-art video generation models produce remarkable photorealism, but they lack the precise control required to align generated content with scene requirements.<n>We present RealMaster, a method that leverages video diffusion models to lift rendered video into photorealistic video while maintaining full alignment with the output of the 3D engine.<n>RealMaster significantly outperforms existing video editing baselines, improving photorealism while preserving the geometry, dynamics, and identity specified by the original 3D control.
  arXiv  Detail & Related papers  (2026-03-24T17:32:42Z)
• IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation [79.1960960864242]
  IllumiCraft is an end-to-end diffusion framework accepting three complementary inputs.<n>It generates temporally coherent videos aligned with user-defined prompts.
  arXiv  Detail & Related papers  (2025-06-03T17:59:52Z)
• VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation [40.16313215714222]
  Controllable image-to-video (I2V) generation transforms a reference image into a coherent video guided by user-specified control signals.<n>We present VidCRAFT3, a unified I2V framework that supports both independent and joint control over camera motion, object motion, and lighting direction.
  arXiv  Detail & Related papers  (2025-02-11T13:11:59Z)
• BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations [82.94002870060045]
  Existing video generation models struggle to follow complex text prompts and synthesize multiple objects.<n>We develop a blob-grounded video diffusion model named BlobGEN-Vid that allows users to control object motions and fine-grained object appearance.<n>We show that our framework is model-agnostic and build BlobGEN-Vid based on both U-Net and DiT-based video diffusion models.
  arXiv  Detail & Related papers  (2025-01-13T19:17:06Z)
• StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models [76.62929629864034]
  We introduce StreetCrafter, a controllable video diffusion model that utilizes LiDAR point cloud renderings as pixel-level conditions.<n>In addition, the utilization of pixel-level LiDAR conditions allows us to make accurate pixel-level edits to target scenes.<n>Our model enables flexible control over viewpoint changes, enlarging the view for satisfying rendering regions.
  arXiv  Detail & Related papers  (2024-12-17T18:58:55Z)
• VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control [74.5434726968562]
  We show how to tame transformers video for 3D camera control using a ControlNet-like conditioning mechanism.<n>Our work is the first to enable camera control for transformer-based video diffusion models.
  arXiv  Detail & Related papers  (2024-07-17T17:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.