RNG: Relightable Neural Gaussians

• URL: http://arxiv.org/abs/2409.19702v3
• Date: Thu, 24 Oct 2024 08:17:05 GMT
• Title: RNG: Relightable Neural Gaussians
• Authors: Jiahui Fan, Fujun Luan, Jian Yang, Miloš Hašan, Beibei Wang,
• Abstract summary: RNG is a novel representation of relightable Gaussians. We propose a depth refinement network to help the shadow under neural 3DGS framework. We achieve about $20times$ faster in training and about $600times$ faster in rendering than prior work.
• Score: 19.197099019727826
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D Gaussian Splatting (3DGS) has shown its impressive power in novel view synthesis. However, creating relightable 3D assets, especially for objects with ill-defined shapes (e.g., fur), is still a challenging task. For these scenes, the decomposition between the light, geometry, and material is more ambiguous, as neither the surface constraints nor the analytical shading model hold. To address this issue, we propose RNG, a novel representation of relightable neural Gaussians, enabling the relighting of objects with both hard surfaces or fluffy boundaries. We avoid any assumptions in the shading model but maintain feature vectors, which can be further decoded by an MLP into colors, in each Gaussian point. Following prior work, we utilize a point light to reduce the ambiguity and introduce a shadow-aware condition to the network. We additionally propose a depth refinement network to help the shadow computation under the 3DGS framework, leading to better shadow effects under point lights. Furthermore, to avoid the blurriness brought by the alpha-blending in 3DGS, we design a hybrid forward-deferred optimization strategy. As a result, we achieve about $20\times$ faster in training and about $600\times$ faster in rendering than prior work based on neural radiance fields, with $60$ frames per second on an RTX4090.

Related papers

• 3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes [87.01284850604495]
  We introduce 3D Convexting (3DCS), which leverages 3D smooth convexes as primitives for modeling geometrically-meaningful radiance fields from multiview images. 3DCS achieves superior performance over 3DGS on benchmarks such as MipNeizer, Tanks and Temples, and Deep Blending. Our results highlight the potential of 3D Convexting to become the new standard for high-quality scene reconstruction.
  arXiv  Detail & Related papers  (2024-11-22T14:31:39Z)
• GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering [83.69136534797686]
  We present GUS-IR, a novel framework designed to address the inverse rendering problem for complicated scenes featuring rough and glossy surfaces. This paper starts by analyzing and comparing two prominent shading techniques popularly used for inverse rendering, forward shading and deferred shading. We propose a unified shading solution that combines the advantages of both techniques for better decomposition.
  arXiv  Detail & Related papers  (2024-11-12T01:51:05Z)
• EVER: Exact Volumetric Ellipsoid Rendering for Real-time View Synthesis [72.53316783628803]
  We present Exact Volumetric Ellipsoid Rendering (EVER), a method for real-time differentiable emission-only volume rendering. Unlike recentization based approach by 3D Gaussian Splatting (3DGS), our primitive based representation allows for exact volume rendering. We show that our method is more accurate with blending issues than 3DGS and follow-up work on view rendering.
  arXiv  Detail & Related papers  (2024-10-02T17:59:09Z)
• GS-ROR: 3D Gaussian Splatting for Reflective Object Relighting via SDF Priors [20.523085632567717]
  We propose GS-ROR for reflective objects relighting with 3DGS aided by SDF priors. Our method outperforms the existing Gaussian-based inverse rendering methods in terms of relighting quality.
  arXiv  Detail & Related papers  (2024-05-22T09:40:25Z)
• DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading [50.331929164207324]
  We introduce DeferredGS, a method for decoupling and editing the Gaussian splatting representation using deferred shading. Both qualitative and quantitative experiments demonstrate the superior performance of DeferredGS in novel view and editing tasks.
  arXiv  Detail & Related papers  (2024-04-15T01:58:54Z)
• Gaussian Shadow Casting for Neural Characters [20.78790953284832]
  We propose a new shadow model using a Gaussian density proxy that replaces sampling with a simple analytic formula. It supports dynamic motion and is tailored for shadow computation, thereby avoiding the affine projection approximation and sorting required by the closely related Gaussian splatting. We demonstrate improved reconstructions, with better separation of albedo, shading, and shadows in challenging outdoor scenes with direct sun light and hard shadows.
  arXiv  Detail & Related papers  (2024-01-11T18:50:31Z)
• Gaussian Splatting with NeRF-based Color and Opacity [7.121259735505479]
  We propose a hybrid model Viewing Direction Gaussian Splatting (VDGS) that uses GS representation of the 3D object's shape and NeRF-based encoding of color and opacity. Our model better describes shadows, light reflections, and the transparency of 3D objects without adding additional texture and light components.
  arXiv  Detail & Related papers  (2023-12-21T10:52:59Z)
• RNA: Relightable Neural Assets [5.955203575139312]
  High-fidelity 3D assets with materials composed of fibers (including hair) are ubiquitous in high-end realistic rendering applications. Shading and scattering models is non-trivial and has to be done not only in the 3D content authoring software, but also in all downstream implementations. We provide an end-to-end shading solution at the first intersection of a ray representation with the underlying geometry.
  arXiv  Detail & Related papers  (2023-12-14T23:37:19Z)
• GS-IR: 3D Gaussian Splatting for Inverse Rendering [71.14234327414086]
  We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) We extend GS, a top-performance representation for novel view synthesis, to estimate scene geometry, surface material, and environment illumination from multi-view images captured under unknown lighting conditions. The flexible and expressive GS representation allows us to achieve fast and compact geometry reconstruction, photorealistic novel view synthesis, and effective physically-based rendering.
  arXiv  Detail & Related papers  (2023-11-26T02:35:09Z)
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis [163.96778522283967]
  We propose a shading-guided generative implicit model that is able to learn a starkly improved shape representation. An accurate 3D shape should also yield a realistic rendering under different lighting conditions. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2021-10-29T10:53:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.