IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination

• URL: http://arxiv.org/abs/2404.11593v2
• Date: Mon, 22 Apr 2024 18:21:24 GMT
• Title: IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
• Authors: Xi Chen, Sida Peng, Dongchen Yang, Yuan Liu, Bowen Pan, Chengfei Lv, Xiaowei Zhou,
• Abstract summary: This paper aims to recover object materials from posed images captured under an unknown static lighting condition. We learn the material prior with a generative model for regularizing the optimization process. Experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery.
• Score: 37.96484120807323
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper aims to recover object materials from posed images captured under an unknown static lighting condition. Recent methods solve this task by optimizing material parameters through differentiable physically based rendering. However, due to the coupling between object geometry, materials, and environment lighting, there is inherent ambiguity during the inverse rendering process, preventing previous methods from obtaining accurate results. To overcome this ill-posed problem, our key idea is to learn the material prior with a generative model for regularizing the optimization process. We observe that the general rendering equation can be split into diffuse and specular shading terms, and thus formulate the material prior as diffusion models of albedo and specular. Thanks to this design, our model can be trained using the existing abundant 3D object data, and naturally acts as a versatile tool to resolve the ambiguity when recovering material representations from RGB images. In addition, we develop a coarse-to-fine training strategy that leverages estimated materials to guide diffusion models to satisfy multi-view consistent constraints, leading to more stable and accurate results. Extensive experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery. The code will be available at https://zju3dv.github.io/IntrinsicAnything.

Related papers

• Material Anything: Generating Materials for Any 3D Object via Diffusion [39.46553064506517]
  We present a fully-automated, unified diffusion framework designed to generate physically-based materials for 3D objects. Material Anything offers a robust, end-to-end solution adaptable to objects under diverse lighting conditions.
  arXiv  Detail & Related papers  (2024-11-22T18:59:39Z)
• MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors [67.74705555889336]
  We introduce MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that incorporates a 2D prior on texture and material properties. We present StableMaterial, a 2D diffusion model prior that refines multi-lit data to estimate the most likely albedo and material from given input appearances. We validate MaterialFusion's relighting performance on 4 datasets of synthetic and real objects under diverse illumination conditions.
  arXiv  Detail & Related papers  (2024-09-23T17:59:06Z)
• A General Albedo Recovery Approach for Aerial Photogrammetric Images through Inverse Rendering [7.874736360019618]
  This paper presents a general image formation model for albedo recovery from typical aerial photogrammetric images under natural illuminations. Our approach builds on the fact that both the sun illumination and scene geometry are estimable in aerial photogrammetry.
  arXiv  Detail & Related papers  (2024-09-04T18:58:32Z)
• MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets [63.284244910964475]
  We propose a 3D asset material generation framework to infer underlying material from the 2D semantic prior. Based on such a prior model, we devise a mechanism to parse material in 3D space.
  arXiv  Detail & Related papers  (2024-04-22T07:00:17Z)
• Intrinsic Image Diffusion for Indoor Single-view Material Estimation [55.276815106443976]
  We present Intrinsic Image Diffusion, a generative model for appearance decomposition of indoor scenes. Given a single input view, we sample multiple possible material explanations represented as albedo, roughness, and metallic maps. Our method produces significantly sharper, more consistent, and more detailed materials, outperforming state-of-the-art methods by $1.5dB$ on PSNR and by $45%$ better FID score on albedo prediction.
  arXiv  Detail & Related papers  (2023-12-19T15:56:19Z)
• Relightify: Relightable 3D Faces from a Single Image via Diffusion Models [86.3927548091627]
  We present the first approach to use diffusion models as a prior for highly accurate 3D facial BRDF reconstruction from a single image. In contrast to existing methods, we directly acquire the observed texture from the input image, thus, resulting in more faithful and consistent estimation.
  arXiv  Detail & Related papers  (2023-05-10T11:57:49Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)
• MaterialGAN: Reflectance Capture using a Generative SVBRDF Model [33.578080406338266]
  We present MaterialGAN, a deep generative convolutional network based on StyleGAN2. We show that MaterialGAN can be used as a powerful material prior in an inverse rendering framework. We demonstrate this framework on the task of reconstructing SVBRDFs from images captured under flash illumination using a hand-held mobile phone.
  arXiv  Detail & Related papers  (2020-09-30T21:33:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.