Material Anything: Generating Materials for Any 3D Object via Diffusion

• URL: http://arxiv.org/abs/2411.15138v1
• Date: Fri, 22 Nov 2024 18:59:39 GMT
• Title: Material Anything: Generating Materials for Any 3D Object via Diffusion
• Authors: Xin Huang, Tengfei Wang, Ziwei Liu, Qing Wang,
• Abstract summary: We present a fully-automated, unified diffusion framework designed to generate physically-based materials for 3D objects. Material Anything offers a robust, end-to-end solution adaptable to objects under diverse lighting conditions.
• Score: 39.46553064506517
• License:
• Abstract: We present Material Anything, a fully-automated, unified diffusion framework designed to generate physically-based materials for 3D objects. Unlike existing methods that rely on complex pipelines or case-specific optimizations, Material Anything offers a robust, end-to-end solution adaptable to objects under diverse lighting conditions. Our approach leverages a pre-trained image diffusion model, enhanced with a triple-head architecture and rendering loss to improve stability and material quality. Additionally, we introduce confidence masks as a dynamic switcher within the diffusion model, enabling it to effectively handle both textured and texture-less objects across varying lighting conditions. By employing a progressive material generation strategy guided by these confidence masks, along with a UV-space material refiner, our method ensures consistent, UV-ready material outputs. Extensive experiments demonstrate our approach outperforms existing methods across a wide range of object categories and lighting conditions.

Related papers

• Boosting 3D Object Generation through PBR Materials [32.732511476490316]
  We propose a novel approach to boost the quality of generated 3D objects from the perspective of Physics-Based Rendering (PBR) materials. For albedo and bump maps, we leverage Stable Diffusion fine-tuned on synthetic data to extract these values. In terms of roughness and metalness maps, we adopt a semi-automatic process to provide room for interactive adjustment.
  arXiv  Detail & Related papers  (2024-11-25T04:20:52Z)
• MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors [67.74705555889336]
  We introduce MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that incorporates a 2D prior on texture and material properties. We present StableMaterial, a 2D diffusion model prior that refines multi-lit data to estimate the most likely albedo and material from given input appearances. We validate MaterialFusion's relighting performance on 4 datasets of synthetic and real objects under diverse illumination conditions.
  arXiv  Detail & Related papers  (2024-09-23T17:59:06Z)
• OpenMaterial: A Comprehensive Dataset of Complex Materials for 3D Reconstruction [54.706361479680055]
  We introduce the OpenMaterial dataset, comprising 1001 objects made of 295 distinct materials. OpenMaterial provides comprehensive annotations, including 3D shape, material type, camera pose, depth, and object mask. It stands as the first large-scale dataset enabling quantitative evaluations of existing algorithms on objects with diverse and challenging materials.
  arXiv  Detail & Related papers  (2024-06-13T07:46:17Z)
• MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets [63.284244910964475]
  We propose a 3D asset material generation framework to infer underlying material from the 2D semantic prior. Based on such a prior model, we devise a mechanism to parse material in 3D space.
  arXiv  Detail & Related papers  (2024-04-22T07:00:17Z)
• IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination [37.96484120807323]
  This paper aims to recover object materials from posed images captured under an unknown static lighting condition. We learn the material prior with a generative model for regularizing the optimization process. Experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery.
  arXiv  Detail & Related papers  (2024-04-17T17:45:08Z)
• ControlMat: A Controlled Generative Approach to Material Capture [7.641962898125423]
  Material reconstruction from a photograph is a key component of 3D content creation democratization. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials.
  arXiv  Detail & Related papers  (2023-09-04T16:18:49Z)
• Relightify: Relightable 3D Faces from a Single Image via Diffusion Models [86.3927548091627]
  We present the first approach to use diffusion models as a prior for highly accurate 3D facial BRDF reconstruction from a single image. In contrast to existing methods, we directly acquire the observed texture from the input image, thus, resulting in more faithful and consistent estimation.
  arXiv  Detail & Related papers  (2023-05-10T11:57:49Z)
• NeILF: Neural Incident Light Field for Physically-based Material Estimation [31.230609753253713]
  We present a differentiable rendering framework for material and lighting estimation from multi-view images and a reconstructed geometry. In the framework, we represent scene lightings as the Neural Incident Light Field (NeILF) and material properties as the surface BRDF modelled by multi-layer perceptrons.
  arXiv  Detail & Related papers  (2022-03-14T15:23:04Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.