Extracting Neural Materials from Multi-view Images

• URL: http://arxiv.org/abs/2606.26715v2
• Date: Fri, 26 Jun 2026 07:21:29 GMT
• Title: Extracting Neural Materials from Multi-view Images
• Abstract summary: We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images.<n>We train a Large Material Reconstruction Model (LMRM) that directly predicts initialbase color, neural material latents, and aleatoric uncertainty guides from images.<n> Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.
• Score: 29.31923332118122
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challenging. We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images. The nonlinear structure of neural material latent spaces makes optimization with naive inverse rendering infeasible. To address this, we train a Large Material Reconstruction Model (LMRM) that directly predicts initialbase color, neural material latents, and aleatoric uncertainty guides from images. This material prior provides a good initialization and better constrains our subsequent optimization using inverse path tracing. The predicted uncertainty further helps by anchoring high-confidence regions more tightly to the LMRM prediction, preventing lighting and complex specular effects from being baked into materials. Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.

Related papers

• Diffusion-Based Material Regularization for Physics-Based Inverse Rendering [15.757134066510135]
  Reconstructing physics-based 3D assets from multi-view images is a core problem in computer graphics and vision.<n>We introduce a regularizer that penalizes deviations in the optimized material over surface regions where the diffusion predictions are near-constant.<n>Our method significantly outperforms state-of-the-art baselines in both reconstruction accuracy and relighting quality.
  arXiv  Detail & Related papers  (2026-06-30T02:52:06Z)
• MatE: Material Extraction from Single-Image via Geometric Prior [36.8533172704247]
  MatE is a novel method for generating tileable PBR materials from a single image taken under unconstrained, real-world conditions.<n>We demonstrate the efficacy and robustness of our approach, enabling users to create realistic materials from real-world image.
  arXiv  Detail & Related papers  (2025-12-20T10:53:49Z)
• Intrinsic Image Fusion for Multi-View 3D Material Reconstruction [49.43509537480623]
  We introduce Intrinsic Image Fusion, a method that reconstructs high-quality physically based materials from multi-view images.<n>Our results outperform state-of-the-art methods in material disentanglement on both synthetic and real scenes.
  arXiv  Detail & Related papers  (2025-12-15T10:05:59Z)
• MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference [83.38607296779423]
  We show that multi-view consistent material inference with more physically-based environment modeling is key to learning accurate reflections with Gaussian Splatting.<n>Our method faithfully recovers both illumination and geometry, achieving state-of-the-art rendering quality in novel views synthesis.
  arXiv  Detail & Related papers  (2025-10-13T13:29:20Z)
• OMG: Opacity Matters in Material Modeling with Gaussian Splatting [4.580002425879323]
  inverse rendering is a long-standing problem in computer vision and graphics.<n>Recent advances in neural rendering enable photo-realistic and plausible inverse rendering results.<n>The emergence of 3D Gaussian Splatting has boosted it to the next level by showing real-time rendering potentials.
  arXiv  Detail & Related papers  (2025-02-16T04:18:41Z)
• OpenMaterial: A Large-scale Dataset of Complex Materials for 3D Reconstruction [55.052637670485716]
  We introduce OpenMaterial, a large-scale semi-synthetic dataset for material-aware 3D reconstruction.<n>It comprises 1,001 objects spanning 295 distinct materials, including conductors, dielectrics, plastics, and their roughened variants, captured under 714 diverse lighting conditions.<n>It provides multi-view images, 3D shape models, camera poses, depth maps, and object masks, establishing the first extensive benchmark for evaluating 3D reconstruction on challenging materials.
  arXiv  Detail & Related papers  (2024-06-13T07:46:17Z)
• IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination [37.96484120807323]
  This paper aims to recover object materials from posed images captured under an unknown static lighting condition. We learn the material prior with a generative model for regularizing the optimization process. Experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery.
  arXiv  Detail & Related papers  (2024-04-17T17:45:08Z)
• Intrinsic Image Diffusion for Indoor Single-view Material Estimation [55.276815106443976]
  We present Intrinsic Image Diffusion, a generative model for appearance decomposition of indoor scenes. Given a single input view, we sample multiple possible material explanations represented as albedo, roughness, and metallic maps. Our method produces significantly sharper, more consistent, and more detailed materials, outperforming state-of-the-art methods by $1.5dB$ on PSNR and by $45%$ better FID score on albedo prediction.
  arXiv  Detail & Related papers  (2023-12-19T15:56:19Z)
• Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering [63.24476194987721]
  Inverse rendering, the process of inferring scene properties from images, is a challenging inverse problem. Most existing solutions incorporate priors into the inverse-rendering pipeline to encourage plausible solutions. We propose a novel scheme that integrates a denoising probabilistic diffusion model pre-trained on natural illumination maps into an optimization framework.
  arXiv  Detail & Related papers  (2023-09-30T12:39:28Z)
• NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination [48.42173911185454]
  Inverse rendering methods aim to estimate geometry, materials and illumination from multi-view RGB images. We propose an end-to-end inverse rendering pipeline that decomposes materials and illumination from multi-view images.
  arXiv  Detail & Related papers  (2023-03-29T12:05:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.