RealMat: Realistic Materials with Diffusion and Reinforcement Learning

• URL: http://arxiv.org/abs/2509.01134v1
• Date: Mon, 01 Sep 2025 05:04:51 GMT
• Title: RealMat: Realistic Materials with Diffusion and Reinforcement Learning
• Authors: Xilong Zhou, Pedro Figueiredo, Miloš Hašan, Valentin Deschaintre, Paul Guerrero, Yiwei Hu, Nima Khademi Kalantari,
• Abstract summary: We propose RealMat, a diffusion-based material generator that leverages realistic priors.<n>We first finetune a pretrained Stable Diffusion XL (SDXL) with synthetic material maps arranged in $2 times 2$ grids.<n>We propose to further finetune our model through reinforcement learning (RL), encouraging the generation of realistic materials.
• Score: 15.780720815063262
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Generative models for high-quality materials are particularly desirable to make 3D content authoring more accessible. However, the majority of material generation methods are trained on synthetic data. Synthetic data provides precise supervision for material maps, which is convenient but also tends to create a significant visual gap with real-world materials. Alternatively, recent work used a small dataset of real flash photographs to guarantee realism, however such data is limited in scale and diversity. To address these limitations, we propose RealMat, a diffusion-based material generator that leverages realistic priors, including a text-to-image model and a dataset of realistic material photos under natural lighting. In RealMat, we first finetune a pretrained Stable Diffusion XL (SDXL) with synthetic material maps arranged in $2 \times 2$ grids. This way, our model inherits some realism of SDXL while learning the data distribution of the synthetic material grids. Still, this creates a realism gap, with some generated materials appearing synthetic. We propose to further finetune our model through reinforcement learning (RL), encouraging the generation of realistic materials. We develop a realism reward function for any material image under natural lighting, by collecting a large-scale dataset of realistic material images. We show that this approach increases generated materials' realism compared to our base model and related work.

Related papers

• VideoNeuMat: Neural Material Extraction from Generative Video Models [8.300347514555337]
  We present VideoNeuMat, a two-stage pipeline that extracts reusable neural material assets from video diffusion models.<n>First, we finetune a large video model to generate material sample videos under controlled camera and lighting trajectories.<n>Second, we reconstruct compact neural materials from these videos through a Large Reconstruction Model (LRM) finetuned from a smaller Wan 1.3B video backbone.
  arXiv  Detail & Related papers  (2026-02-06T23:49:10Z)
• Large Material Gaussian Model for Relightable 3D Generation [54.10879517395551]
  We introduce a novel framework designed to generate high-quality 3D content with Physically Based Rendering (PBR) materials.<n>Our method not only exhibit greater visual appeal compared to baseline methods but also enhance material modeling, thereby enabling practical downstream rendering applications.
  arXiv  Detail & Related papers  (2025-09-26T09:35:12Z)
• MatSwap: Light-aware material transfers in images [18.37330769828654]
  MatSwap is a method to transfer materials to designated surfaces in an image photorealistically.<n>We learn the relationship between the input material and its appearance within the scene, without the need for explicit UV mapping.<n>Our method seamlessly integrates a desired material into the target location in the photograph while retaining the identity of the scene.
  arXiv  Detail & Related papers  (2025-02-11T18:59:59Z)
• MaPa: Text-driven Photorealistic Material Painting for 3D Shapes [79.13775179541311]
  This paper aims to generate materials for 3D meshes from text descriptions.<n>Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs.<n>Our framework supports high-quality rendering and provides substantial flexibility in editing.
  arXiv  Detail & Related papers  (2024-04-26T17:54:38Z)
• Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials [108.59709545364395]
  GPT-4V can effectively recognize and describe materials, allowing the construction of a detailed material library. The correctly matched materials are then meticulously applied as reference for the new SVBRDF material generation. Make-it-Real offers a streamlined integration into the 3D content creation workflow.
  arXiv  Detail & Related papers  (2024-04-25T17:59:58Z)
• Intrinsic Image Diffusion for Indoor Single-view Material Estimation [55.276815106443976]
  We present Intrinsic Image Diffusion, a generative model for appearance decomposition of indoor scenes. Given a single input view, we sample multiple possible material explanations represented as albedo, roughness, and metallic maps. Our method produces significantly sharper, more consistent, and more detailed materials, outperforming state-of-the-art methods by $1.5dB$ on PSNR and by $45%$ better FID score on albedo prediction.
  arXiv  Detail & Related papers  (2023-12-19T15:56:19Z)
• Alchemist: Parametric Control of Material Properties with Diffusion Models [51.63031820280475]
  Our method capitalizes on the generative prior of text-to-image models known for photorealism. We show the potential application of our model to material edited NeRFs.
  arXiv  Detail & Related papers  (2023-12-05T18:58:26Z)
• PhotoMat: A Material Generator Learned from Single Flash Photos [37.42765147463852]
  Previous generative models for materials have been trained exclusively on synthetic data. We propose PhotoMat: the first material generator trained exclusively on real photos of material samples captured using a cell phone camera with flash. We show that our generated materials have better visual quality than previous material generators trained on synthetic data.
  arXiv  Detail & Related papers  (2023-05-20T22:27:41Z)
• OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets [103.54691385842314]
  We propose a novel framework for creating large-scale photorealistic datasets of indoor scenes. Our goal is to make the dataset creation process widely accessible. This enables important applications in inverse rendering, scene understanding and robotics.
  arXiv  Detail & Related papers  (2020-07-25T06:48:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.