MatFuse: Controllable Material Generation with Diffusion Models

• URL: http://arxiv.org/abs/2308.11408v3
• Date: Wed, 13 Mar 2024 10:31:21 GMT
• Title: MatFuse: Controllable Material Generation with Diffusion Models
• Authors: Giuseppe Vecchio, Renato Sortino, Simone Palazzo, Concetto Spampinato
• Abstract summary: MatFuse is a unified approach that harnesses the generative power of diffusion models for creation and editing of 3D materials. Our method integrates multiple sources of conditioning, including color palettes, sketches, text, and pictures, enhancing creative possibilities. We demonstrate the effectiveness of MatFuse under multiple conditioning settings and explore the potential of material editing.
• Score: 10.993516790237503
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Creating high-quality materials in computer graphics is a challenging and time-consuming task, which requires great expertise. To simplify this process, we introduce MatFuse, a unified approach that harnesses the generative power of diffusion models for creation and editing of 3D materials. Our method integrates multiple sources of conditioning, including color palettes, sketches, text, and pictures, enhancing creative possibilities and granting fine-grained control over material synthesis. Additionally, MatFuse enables map-level material editing capabilities through latent manipulation by means of a multi-encoder compression model which learns a disentangled latent representation for each map. We demonstrate the effectiveness of MatFuse under multiple conditioning settings and explore the potential of material editing. Finally, we assess the quality of the generated materials both quantitatively in terms of CLIP-IQA and FID scores and qualitatively by conducting a user study. Source code for training MatFuse and supplemental materials are publicly available at https://gvecchio.com/matfuse.

Related papers

• From Tokens to Materials: Leveraging Language Models for Scientific Discovery [12.211984932142537]
  This study investigates the application of language model embeddings to enhance material property prediction in materials science. We demonstrate that domain-specific models, particularly MatBERT, significantly outperform general-purpose models in extracting implicit knowledge from compound names and material properties.
  arXiv  Detail & Related papers  (2024-10-21T16:31:23Z)
• MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors [67.74705555889336]
  We introduce MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that incorporates a 2D prior on texture and material properties. We present StableMaterial, a 2D diffusion model prior that refines multi-lit data to estimate the most likely albedo and material from given input appearances. We validate MaterialFusion's relighting performance on 4 datasets of synthetic and real objects under diverse illumination conditions.
  arXiv  Detail & Related papers  (2024-09-23T17:59:06Z)
• MaPa: Text-driven Photorealistic Material Painting for 3D Shapes [80.66880375862628]
  This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs. Our framework supports high-quality rendering and provides substantial flexibility in editing.
  arXiv  Detail & Related papers  (2024-04-26T17:54:38Z)
• Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials [108.59709545364395]
  GPT-4V can effectively recognize and describe materials, allowing the construction of a detailed material library. The correctly matched materials are then meticulously applied as reference for the new SVBRDF material generation. Make-it-Real offers a streamlined integration into the 3D content creation workflow.
  arXiv  Detail & Related papers  (2024-04-25T17:59:58Z)
• MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets [63.284244910964475]
  We propose a 3D asset material generation framework to infer underlying material from the 2D semantic prior. Based on such a prior model, we devise a mechanism to parse material in 3D space.
  arXiv  Detail & Related papers  (2024-04-22T07:00:17Z)
• Alchemist: Parametric Control of Material Properties with Diffusion Models [51.63031820280475]
  Our method capitalizes on the generative prior of text-to-image models known for photorealism. We show the potential application of our model to material edited NeRFs.
  arXiv  Detail & Related papers  (2023-12-05T18:58:26Z)
• Multimodal Learning for Materials [7.167520424757711]
  We introduce Multimodal Learning for Materials (MultiMat), which enables self-supervised multi-modality training of foundation models for materials. We demonstrate our framework's potential using data from the Materials Project database on multiple axes.
  arXiv  Detail & Related papers  (2023-11-30T18:35:29Z)
• One-shot recognition of any material anywhere using contrastive learning with physics-based rendering [0.0]
  We present MatSim: a synthetic dataset, a benchmark, and a method for computer vision based recognition of similarities and transitions between materials and textures. The visual recognition of materials is essential to everything from examining food while cooking to inspecting agriculture, chemistry, and industrial products.
  arXiv  Detail & Related papers  (2022-12-01T16:49:53Z)
• MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction [13.924666106089425]
  MatSciBERT is a language model trained on a large corpus of scientific literature published in the materials domain. We show that MatSciBERT outperforms SciBERT on three downstream tasks, namely, abstract classification, named entity recognition, and relation extraction. We also discuss some of the applications of MatSciBERT in the materials domain for extracting information.
  arXiv  Detail & Related papers  (2021-09-30T17:35:02Z)
• Deep Video Matting via Spatio-Temporal Alignment and Aggregation [63.6870051909004]
  We propose a deep learning-based video matting framework which employs a novel aggregation feature module (STFAM) To eliminate frame-by-frame trimap annotations, a lightweight interactive trimap propagation network is also introduced. Our framework significantly outperforms conventional video matting and deep image matting methods.
  arXiv  Detail & Related papers  (2021-04-22T17:42:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.