RNA: Relightable Neural Assets

• URL: http://arxiv.org/abs/2312.09398v2
• Date: Mon, 16 Sep 2024 18:41:04 GMT
• Title: RNA: Relightable Neural Assets
• Authors: Krishna Mullia, Fujun Luan, Xin Sun, Miloš Hašan,
• Abstract summary: High-fidelity 3D assets with materials composed of fibers (including hair) are ubiquitous in high-end realistic rendering applications. Shading and scattering models is non-trivial and has to be done not only in the 3D content authoring software, but also in all downstream implementations. We provide an end-to-end shading solution at the first intersection of a ray representation with the underlying geometry.
• Score: 5.955203575139312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-fidelity 3D assets with materials composed of fibers (including hair), complex layered material shaders, or fine scattering geometry are ubiquitous in high-end realistic rendering applications. Rendering such models is computationally expensive due to heavy shaders and long scattering paths. Moreover, implementing the shading and scattering models is non-trivial and has to be done not only in the 3D content authoring software (which is necessarily complex), but also in all downstream rendering solutions. For example, web and mobile viewers for complex 3D assets are desirable, but frequently cannot support the full shading complexity allowed by the authoring application. Our goal is to design a neural representation for 3D assets with complex shading that supports full relightability and full integration into existing renderers. We provide an end-to-end shading solution at the first intersection of a ray with the underlying geometry. All shading and scattering is precomputed and included in the neural asset; no multiple scattering paths need to be traced, and no complex shading models need to be implemented to render our assets, beyond a single neural architecture. We combine an MLP decoder with a feature grid. Shading consists of querying a feature vector, followed by an MLP evaluation producing the final reflectance value. Our method provides high-fidelity shading, close to the ground-truth Monte Carlo estimate even at close-up views. We believe our neural assets could be used in practical renderers, providing significant speed-ups and simplifying renderer implementations.

Related papers

• GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering [83.69136534797686]
  We present GUS-IR, a novel framework designed to address the inverse rendering problem for complicated scenes featuring rough and glossy surfaces. This paper starts by analyzing and comparing two prominent shading techniques popularly used for inverse rendering, forward shading and deferred shading. We propose a unified shading solution that combines the advantages of both techniques for better decomposition.
  arXiv  Detail & Related papers  (2024-11-12T01:51:05Z)
• RelitLRM: Generative Relightable Radiance for Large Reconstruction Models [52.672706620003765]
  We propose RelitLRM for generating high-quality Gaussian splatting representations of 3D objects under novel illuminations. Unlike prior inverse rendering methods requiring dense captures and slow optimization, RelitLRM adopts a feed-forward transformer-based model. We show our sparse-view feed-forward RelitLRM offers competitive relighting results to state-of-the-art dense-view optimization-based baselines.
  arXiv  Detail & Related papers  (2024-10-08T17:40:01Z)
• RNG: Relightable Neural Gaussians [19.197099019727826]
  RNG is a novel representation of relightable Gaussians. We propose a depth refinement network to help the shadow under neural 3DGS framework. We achieve about $20times$ faster in training and about $600times$ faster in rendering than prior work.
  arXiv  Detail & Related papers  (2024-09-29T13:32:24Z)
• Neural Assets: Volumetric Object Capture and Rendering for Interactive Environments [8.258451067861932]
  We propose an approach for capturing real-world objects in everyday environments faithfully and fast. We use a novel neural representation to reconstruct effects, such as translucent object parts, and preserve object appearance. This leads to a seamless integration of the proposed neural assets with existing mesh environments and objects.
  arXiv  Detail & Related papers  (2022-12-12T18:55:03Z)
• Multi-View Mesh Reconstruction with Neural Deferred Shading [0.8514420632209809]
  State-of-the-art methods use both neural surface representations and neural shading. We represent surfaces as triangle meshes and build a differentiable rendering pipeline around triangle rendering and neural shading. We evaluate our runtime on a public 3D reconstruction dataset and show that it can match the reconstruction accuracy of traditional baselines while surpassing them in optimization.
  arXiv  Detail & Related papers  (2022-12-08T16:29:46Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• 3D Scene Creation and Rendering via Rough Meshes: A Lighting Transfer Avenue [49.62477229140788]
  This paper studies how to flexibly integrate reconstructed 3D models into practical 3D modeling pipelines such as 3D scene creation and rendering. We propose a lighting transfer network (LighTNet) to bridge NFR and PBR, such that they can benefit from each other.
  arXiv  Detail & Related papers  (2022-11-27T13:31:00Z)
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis [163.96778522283967]
  We propose a shading-guided generative implicit model that is able to learn a starkly improved shape representation. An accurate 3D shape should also yield a realistic rendering under different lighting conditions. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2021-10-29T10:53:12Z)
• Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
  We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
  arXiv  Detail & Related papers  (2021-09-13T15:29:03Z)
• Shape From Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF Material from Images via Differentiable Path Tracing [16.975014467319443]
  Differentiable path tracing is an appealing framework as it can reproduce complex appearance effects. We show how to use differentiable ray tracing to refine an initial coarse mesh and per-mesh-facet material representation. We also show how to refine initial reconstructions of real-world objects in unconstrained environments.
  arXiv  Detail & Related papers  (2020-12-06T18:55:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.