Material Transforms from Disentangled NeRF Representations

• URL: http://arxiv.org/abs/2411.08037v1
• Date: Tue, 12 Nov 2024 18:59:59 GMT
• Title: Material Transforms from Disentangled NeRF Representations
• Authors: Ivan Lopes, Jean-François Lalonde, Raoul de Charette,
• Abstract summary: We propose a novel method for transferring material transformations across different scenes. We learn to map Bidirectional Reflectance Distribution Functions (BRDF) from pairs of scenes observed in varying conditions. The learned transformations can then be applied to unseen scenes with similar materials, therefore effectively rendering the transformation learned with an arbitrary level of intensity.
• Score: 23.688782106067166
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we first propose a novel method for transferring material transformations across different scenes. Building on disentangled Neural Radiance Field (NeRF) representations, our approach learns to map Bidirectional Reflectance Distribution Functions (BRDF) from pairs of scenes observed in varying conditions, such as dry and wet. The learned transformations can then be applied to unseen scenes with similar materials, therefore effectively rendering the transformation learned with an arbitrary level of intensity. Extensive experiments on synthetic scenes and real-world objects validate the effectiveness of our approach, showing that it can learn various transformations such as wetness, painting, coating, etc. Our results highlight not only the versatility of our method but also its potential for practical applications in computer graphics. We publish our method implementation, along with our synthetic/real datasets on https://github.com/astra-vision/BRDFTransform

Related papers

• Materialist: Physically Based Editing Using Single-Image Inverse Rendering [50.39048790589746]
  We present a method combining a learning-based approach with progressive differentiable rendering. Our method achieves more realistic light material interactions, accurate shadows, and global illumination. We also propose a method for material transparency editing that operates effectively without requiring full scene geometry.
  arXiv  Detail & Related papers  (2025-01-07T11:52:01Z)
• Triplet: Triangle Patchlet for Mesh-Based Inverse Rendering and Scene Parameters Approximation [0.0]
  inverse rendering seeks to derive the physical properties of a scene, including light, geometry, textures, and materials. Meshes, as a traditional representation adopted by many simulation pipeline, still show limited influence in radiance field for inverse rendering. This paper introduces a novel framework called Triangle Patchlet (abbr. Triplet), a mesh-based representation, to comprehensively approximate these parameters.
  arXiv  Detail & Related papers  (2024-10-16T09:59:11Z)
• Learning Relighting and Intrinsic Decomposition in Neural Radiance Fields [21.057216351934688]
  Our research introduces a method that combines relighting with intrinsic decomposition. By leveraging light variations in scenes to generate pseudo labels, our method provides guidance for intrinsic decomposition. We validate our method on both synthetic and real-world datasets, achieving convincing results.
  arXiv  Detail & Related papers  (2024-06-16T21:10:37Z)
• Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering [63.24476194987721]
  Inverse rendering, the process of inferring scene properties from images, is a challenging inverse problem. Most existing solutions incorporate priors into the inverse-rendering pipeline to encourage plausible solutions. We propose a novel scheme that integrates a denoising probabilistic diffusion model pre-trained on natural illumination maps into an optimization framework.
  arXiv  Detail & Related papers  (2023-09-30T12:39:28Z)
• Neural Relighting with Subsurface Scattering by Learning the Radiance Transfer Gradient [73.52585139592398]
  We propose a novel framework for learning the radiance transfer field via volume rendering. We will release our code and a novel light stage dataset of objects with subsurface scattering effects publicly available.
  arXiv  Detail & Related papers  (2023-06-15T17:56:04Z)
• NeuS-PIR: Learning Relightable Neural Surface using Pre-Integrated Rendering [23.482941494283978]
  This paper presents a method, namely NeuS-PIR, for recovering relightable neural surfaces from multi-view images or video. Unlike methods based on NeRF and discrete meshes, our method utilizes implicit neural surface representation to reconstruct high-quality geometry. Our method enables advanced applications such as relighting, which can be seamlessly integrated with modern graphics engines.
  arXiv  Detail & Related papers  (2023-06-13T09:02:57Z)
• Physics-based Indirect Illumination for Inverse Rendering [70.27534648770057]
  We present a physics-based inverse rendering method that learns the illumination, geometry, and materials of a scene from posed multi-view RGB images. As a side product, our physics-based inverse rendering model also facilitates flexible and realistic material editing as well as relighting.
  arXiv  Detail & Related papers  (2022-12-09T07:33:49Z)
• RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation [110.4255414234771]
  Existing solutions require massive training data or lack generalizability to unknown rendering configurations. We propose a novel approach that marries domain randomization and differentiable rendering gradients to address this problem. Our approach achieves significantly lower reconstruction errors and has better generalizability among unknown rendering configurations.
  arXiv  Detail & Related papers  (2022-05-11T17:59:51Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)
• Partially fake it till you make it: mixing real and fake thermal images for improved object detection [29.13557322147509]
  We show the performance of the proposed system in the context of object detection in thermal videos. Our single-modality detector achieves state-of-the-art results on the FLIR ADAS dataset.
  arXiv  Detail & Related papers  (2021-06-25T12:56:09Z)
• PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting [60.75436852495868]
  We present PhySG, an inverse rendering pipeline that reconstructs geometry, materials, and illumination from scratch from RGB input images. We demonstrate, with both synthetic and real data, that our reconstructions not only enable rendering of novel viewpoints, but also physics-based appearance editing of materials and illumination.
  arXiv  Detail & Related papers  (2021-04-01T17:59:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.