Diffusion-Based Material Regularization for Physics-Based Inverse Rendering

• URL: http://arxiv.org/abs/2606.31065v1
• Date: Tue, 30 Jun 2026 02:52:06 GMT
• Title: Diffusion-Based Material Regularization for Physics-Based Inverse Rendering
• Abstract summary: Reconstructing physics-based 3D assets from multi-view images is a core problem in computer graphics and vision.<n>We introduce a regularizer that penalizes deviations in the optimized material over surface regions where the diffusion predictions are near-constant.<n>Our method significantly outperforms state-of-the-art baselines in both reconstruction accuracy and relighting quality.
• Score: 15.757134066510135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing physics-based 3D assets -- geometry, materials, and illumination -- from multi-view images is a core problem in computer graphics and vision, and a prerequisite for realistic relighting and editing. Physics-based inverse rendering offers an accurate image-formation model, but is severely underconstrained: without strong priors, illumination is baked into materials, and reconstructions generalize poorly to novel views and lighting. Data-driven diffusion models, in contrast, predict visually plausible materials, yet their predictions rarely satisfy the rendering equation and are not directly usable for physics-based rendering. We bridge these two paradigms rather than replacing either. Our key idea is to treat the predictions of a state-of-the-art diffusion model not as target material values but as a similarity kernel for optimization: we introduce a regularization loss that penalizes deviations in the optimized material over surface regions where the diffusion predictions are near-constant, while leaving the optimization free to match the input images. Built on this regularizer, our end-to-end pipeline jointly reconstructs geometry, materials, and illumination, yielding high-quality assets that drop into standard rendering pipelines and relight faithfully. On the Synthetic4Relight, Stanford-ORB, and DTC-Synthetic datasets, our method significantly outperforms state-of-the-art baselines in both reconstruction accuracy and relighting quality.

Related papers

• MatE: Material Extraction from Single-Image via Geometric Prior [36.8533172704247]
  MatE is a novel method for generating tileable PBR materials from a single image taken under unconstrained, real-world conditions.<n>We demonstrate the efficacy and robustness of our approach, enabling users to create realistic materials from real-world image.
  arXiv  Detail & Related papers  (2025-12-20T10:53:49Z)
• Intrinsic Image Fusion for Multi-View 3D Material Reconstruction [49.43509537480623]
  We introduce Intrinsic Image Fusion, a method that reconstructs high-quality physically based materials from multi-view images.<n>Our results outperform state-of-the-art methods in material disentanglement on both synthetic and real scenes.
  arXiv  Detail & Related papers  (2025-12-15T10:05:59Z)
• MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference [83.38607296779423]
  We show that multi-view consistent material inference with more physically-based environment modeling is key to learning accurate reflections with Gaussian Splatting.<n>Our method faithfully recovers both illumination and geometry, achieving state-of-the-art rendering quality in novel views synthesis.
  arXiv  Detail & Related papers  (2025-10-13T13:29:20Z)
• OMG: Opacity Matters in Material Modeling with Gaussian Splatting [4.580002425879323]
  inverse rendering is a long-standing problem in computer vision and graphics.<n>Recent advances in neural rendering enable photo-realistic and plausible inverse rendering results.<n>The emergence of 3D Gaussian Splatting has boosted it to the next level by showing real-time rendering potentials.
  arXiv  Detail & Related papers  (2025-02-16T04:18:41Z)
• DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models [83.28670336340608]
  We introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering.<n>Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.
  arXiv  Detail & Related papers  (2025-01-30T18:59:11Z)
• PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields [49.6405458373509]
  We present an inverse rendering (IR) model capable of jointly estimating scene geometry, materials, and illumination.<n>Our method is easily adaptable to other inverse rendering and 3D reconstruction frameworks that require material estimation.
  arXiv  Detail & Related papers  (2024-12-12T19:00:21Z)
• RelitLRM: Generative Relightable Radiance for Large Reconstruction Models [52.672706620003765]
  We propose RelitLRM for generating high-quality Gaussian splatting representations of 3D objects under novel illuminations. Unlike prior inverse rendering methods requiring dense captures and slow optimization, RelitLRM adopts a feed-forward transformer-based model. We show our sparse-view feed-forward RelitLRM offers competitive relighting results to state-of-the-art dense-view optimization-based baselines.
  arXiv  Detail & Related papers  (2024-10-08T17:40:01Z)
• IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination [37.96484120807323]
  This paper aims to recover object materials from posed images captured under an unknown static lighting condition. We learn the material prior with a generative model for regularizing the optimization process. Experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery.
  arXiv  Detail & Related papers  (2024-04-17T17:45:08Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.