MatFuse: Controllable Material Generation with Diffusion Models
- URL: http://arxiv.org/abs/2308.11408v3
- Date: Wed, 13 Mar 2024 10:31:21 GMT
- Title: MatFuse: Controllable Material Generation with Diffusion Models
- Authors: Giuseppe Vecchio, Renato Sortino, Simone Palazzo, Concetto Spampinato
- Abstract summary: MatFuse is a unified approach that harnesses the generative power of diffusion models for creation and editing of 3D materials.
Our method integrates multiple sources of conditioning, including color palettes, sketches, text, and pictures, enhancing creative possibilities.
We demonstrate the effectiveness of MatFuse under multiple conditioning settings and explore the potential of material editing.
- Score: 10.993516790237503
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Creating high-quality materials in computer graphics is a challenging and
time-consuming task, which requires great expertise. To simplify this process,
we introduce MatFuse, a unified approach that harnesses the generative power of
diffusion models for creation and editing of 3D materials. Our method
integrates multiple sources of conditioning, including color palettes,
sketches, text, and pictures, enhancing creative possibilities and granting
fine-grained control over material synthesis. Additionally, MatFuse enables
map-level material editing capabilities through latent manipulation by means of
a multi-encoder compression model which learns a disentangled latent
representation for each map. We demonstrate the effectiveness of MatFuse under
multiple conditioning settings and explore the potential of material editing.
Finally, we assess the quality of the generated materials both quantitatively
in terms of CLIP-IQA and FID scores and qualitatively by conducting a user
study. Source code for training MatFuse and supplemental materials are publicly
available at https://gvecchio.com/matfuse.
Related papers
- MaPa: Text-driven Photorealistic Material Painting for 3D Shapes [80.66880375862628]
This paper aims to generate materials for 3D meshes from text descriptions.
Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs.
Our framework supports high-quality rendering and provides substantial flexibility in editing.
arXiv Detail & Related papers (2024-04-26T17:54:38Z) - Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials [108.59709545364395]
GPT-4V can effectively recognize and describe materials, allowing the construction of a detailed material library.
The correctly matched materials are then meticulously applied as reference for the new SVBRDF material generation.
Make-it-Real offers a streamlined integration into the 3D content creation workflow.
arXiv Detail & Related papers (2024-04-25T17:59:58Z) - MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets [63.284244910964475]
We propose a 3D asset material generation framework to infer underlying material from the 2D semantic prior.
Based on such a prior model, we devise a mechanism to parse material in 3D space.
arXiv Detail & Related papers (2024-04-22T07:00:17Z) - Alchemist: Parametric Control of Material Properties with Diffusion
Models [51.63031820280475]
Our method capitalizes on the generative prior of text-to-image models known for photorealism.
We show the potential application of our model to material edited NeRFs.
arXiv Detail & Related papers (2023-12-05T18:58:26Z) - Multimodal Learning for Materials [7.167520424757711]
We introduce Multimodal Learning for Materials (MultiMat), which enables self-supervised multi-modality training of foundation models for materials.
We demonstrate our framework's potential using data from the Materials Project database on multiple axes.
arXiv Detail & Related papers (2023-11-30T18:35:29Z) - One-shot recognition of any material anywhere using contrastive learning
with physics-based rendering [0.0]
We present MatSim: a synthetic dataset, a benchmark, and a method for computer vision based recognition of similarities and transitions between materials and textures.
The visual recognition of materials is essential to everything from examining food while cooking to inspecting agriculture, chemistry, and industrial products.
arXiv Detail & Related papers (2022-12-01T16:49:53Z) - MatSciBERT: A Materials Domain Language Model for Text Mining and
Information Extraction [13.924666106089425]
MatSciBERT is a language model trained on a large corpus of scientific literature published in the materials domain.
We show that MatSciBERT outperforms SciBERT on three downstream tasks, namely, abstract classification, named entity recognition, and relation extraction.
We also discuss some of the applications of MatSciBERT in the materials domain for extracting information.
arXiv Detail & Related papers (2021-09-30T17:35:02Z) - Attention-guided Temporal Coherent Video Object Matting [78.82835351423383]
We propose a novel deep learning-based object matting method that can achieve temporally coherent matting results.
Its key component is an attention-based temporal aggregation module that maximizes image matting networks' strength.
We show how to effectively solve the trimap generation problem by fine-tuning a state-of-the-art video object segmentation network.
arXiv Detail & Related papers (2021-05-24T17:34:57Z) - Deep Video Matting via Spatio-Temporal Alignment and Aggregation [63.6870051909004]
We propose a deep learning-based video matting framework which employs a novel aggregation feature module (STFAM)
To eliminate frame-by-frame trimap annotations, a lightweight interactive trimap propagation network is also introduced.
Our framework significantly outperforms conventional video matting and deep image matting methods.
arXiv Detail & Related papers (2021-04-22T17:42:08Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.