Relightable Gaussian Splatting for Virtual Production Using Image-Based Illumination

• URL: http://arxiv.org/abs/2605.09024v1
• Date: Sat, 09 May 2026 16:04:06 GMT
• Title: Relightable Gaussian Splatting for Virtual Production Using Image-Based Illumination
• Abstract summary: Virtual production (VP) use LED walls to provide both background imagery and image-based lighting.<n>In inverse rendering, physically-based rendering relies on estimates 3D geometry and lighting, using environment maps.<n>We propose a VP-specific framework for 3D reconstruction and relighting using Gaussian Splatting.
• Score: 5.673030658112764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Virtual production (VP) use LED walls to provide both background imagery and image-based lighting. While this enables on-set compositing, it couples lighting to background and scene appearance, limiting flexibility for downstream editing. In addition, inverse rendering conventionally relies on physically-based rendering to estimates 3D geometry and lighting, using environment maps. However, these maps are typically low-resolution and assume far-field lighting. In VP, with near-field and high-resolution image-based lighting, this can lead to inaccuracies and introduce complexities when editing. Addressing this, we propose a VP-specific framework for 3D reconstruction and relighting using Gaussian Splatting. This uses the known background imagery to condition the relighting process. This avoids relying on environment maps and reduces compositing to a background-image editing task. To realize our framework, we introduce a process (and associated dataset) that captures real VP scenes under varying background content and illumination conditions. This data is used to decompose a 3D scene into fixed appearance and variable lighting components. The variable lighting process simulates light transport by parameterizing each primitive with a UV coordinate, intensity value and resolution modifier. Using mipmaps, these directly sample the background texture in image space - implicitly capturing reflections and refractions without physically-based rendering. Combined with the fixed appearance component, this allows us to render relit scenes using a Gaussian Splatting rasterizer. Compared to baselines, our approach achieves higher-quality 3D reconstruction and controllable relighting. The method is efficient (<3 GB RAM, <5 GB VRAM, <2 hours training, ~35 FPS) and supports rendering useful arbitrary output variables including depth, lighting intensity, lighting color, and unlit renders.

Related papers

• LightFuse: Relightable Interactive Gaussian Scene Reconstruction via Multi-Scan Fusion and 2D Gaussian Ray Tracing [19.013915783052433]
  Relightable interactive scene reconstruction aims to build an editable 3D model from scans of different object arrangements and render new layouts under novel illumination.<n>We present LightFuse, a 2D Gaussian framework that extends interactive scene reconstruction with explicit material-illumination decomposition and physically based relighting.
  arXiv  Detail & Related papers  (2026-08-29T13:49:13Z)
• PIXLRelight: Controllable Relighting via Intrinsic Conditioning [9.829003409973486]
  We present PIXLRelight, a feed-forward approach for physically controllable single-image relighting.<n> PIXLRelight enables arbitrary PBR-style lighting control, achieves state-of-the-art relighting quality, and runs in under a tenth of a second per image.
  arXiv  Detail & Related papers  (2026-05-18T17:55:03Z)
• ComGS: Efficient 3D Object-Scene Composition via Surface Octahedral Probes [46.83857963152283]
  Gaussian Splatting (GS) enables immersive rendering, but realistic 3D object-scene composition remains challenging.<n>We propose ComGS, a novel 3D object-scene composition framework.<n>Our method achieves high-quality, real-time rendering at around 28 FPS, produces visually harmonious results with vivid shadows, and requires only 36 seconds for editing.
  arXiv  Detail & Related papers  (2025-10-09T03:10:41Z)
• 3D Gaussian Inverse Rendering with Approximated Global Illumination [15.899514468603627]
  We present a novel approach that enables efficient global illumination for 3D Gaussians Splatting through screen-space ray tracing.<n>Our key insight is that a substantial amount of indirect light can be traced back to surfaces visible within the current view frustum.<n>In experiments, we show that the screen-space approximation we utilize allows for indirect illumination and supports real-time rendering and editing.
  arXiv  Detail & Related papers  (2025-04-02T05:02:25Z)
• EVER: Exact Volumetric Ellipsoid Rendering for Real-time View Synthesis [72.53316783628803]
  We present Exact Volumetric Ellipsoid Rendering (EVER), a method for real-time differentiable emission-only volume rendering.<n>Unlike recentization based approach by 3D Gaussian Splatting (3DGS), our primitive based representation allows for exact volume rendering.<n>We show that our method is more accurate with blending issues than 3DGS and follow-up work on view rendering.
  arXiv  Detail & Related papers  (2024-10-02T17:59:09Z)
• DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading [50.331929164207324]
  We introduce DeferredGS, a method for decoupling and editing the Gaussian splatting representation using deferred shading. Both qualitative and quantitative experiments demonstrate the superior performance of DeferredGS in novel view and editing tasks.
  arXiv  Detail & Related papers  (2024-04-15T01:58:54Z)
• Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing [21.498078188364566]
  We present a novel differentiable point-based rendering framework to achieve photo-realistic relighting. The proposed framework showcases the potential to revolutionize the mesh-based graphics pipeline with a point-based pipeline enabling editing, tracing, and relighting.
  arXiv  Detail & Related papers  (2023-11-27T18:07:58Z)
• Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes [62.769186261245416]
  We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows.
  arXiv  Detail & Related papers  (2023-04-06T17:51:54Z)
• Physically-Based Editing of Indoor Scene Lighting from a Single Image [106.60252793395104]
  We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions.
  arXiv  Detail & Related papers  (2022-05-19T06:44:37Z)
• Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
  We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
  arXiv  Detail & Related papers  (2021-09-13T15:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.