VideoMatGen: PBR Materials through Joint Generative Modeling

• URL: http://arxiv.org/abs/2603.16566v1
• Date: Tue, 17 Mar 2026 14:24:20 GMT
• Title: VideoMatGen: PBR Materials through Joint Generative Modeling
• Abstract summary: We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture.<n>Our method is conditioned on input geometry and a text description, and jointly models multiple material properties.<n>We introduce a custom variational auto-encoder which encodes multiple material modalities into a compact latent space.
• Score: 11.245666691622459
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a method for generating physically-based materials for 3D shapes based on a video diffusion transformer architecture. Our method is conditioned on input geometry and a text description, and jointly models multiple material properties (base color, roughness, metallicity, height map) to form physically plausible materials. We further introduce a custom variational auto-encoder which encodes multiple material modalities into a compact latent space, which enables joint generation of multiple modalities without increasing the number of tokens. Our pipeline generates high-quality materials for 3D shapes given a text prompt, compatible with common content creation tools.

Related papers

• Large Material Gaussian Model for Relightable 3D Generation [54.10879517395551]
  We introduce a novel framework designed to generate high-quality 3D content with Physically Based Rendering (PBR) materials.<n>Our method not only exhibit greater visual appeal compared to baseline methods but also enhance material modeling, thereby enabling practical downstream rendering applications.
  arXiv  Detail & Related papers  (2025-09-26T09:35:12Z)
• VideoMat: Extracting PBR Materials from Video Diffusion Models [11.48114859355725]
  We leverage finetuned video diffusion models, intrinsic decomposition of videos, and physically-based differentiable rendering to generate high quality materials for 3D models given a text prompt or a single image.
  arXiv  Detail & Related papers  (2025-06-11T12:36:49Z)
• Pandora3D: A Comprehensive Framework for High-Quality 3D Shape and Texture Generation [56.862552362223425]
  This report presents a comprehensive framework for generating high-quality 3D shapes and textures from diverse input prompts.<n>The framework consists of 3D shape generation and texture generation.<n>This report details the system architecture, experimental results, and potential future directions to improve and expand the framework.
  arXiv  Detail & Related papers  (2025-02-20T04:22:30Z)
• MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Models [42.42328559042189]
  MatCLIP is a novel method that extracts shape- and lighting-insensitive descriptors of PBR materials to assign plausible textures to 3D objects based on images.<n>By extending an Alpha-CLIP-based model on material renderings across diverse shapes and lighting, our approach generates descriptors that bridge the domains of PBR representations with photographs or renderings.<n>MatCLIP achieves a top-1 classification accuracy of 76.6%, outperforming state-of-the-art methods such as PhotoShape and MatAtlas.
  arXiv  Detail & Related papers  (2025-01-27T12:08:52Z)
• ArtFormer: Controllable Generation of Diverse 3D Articulated Objects [5.320860732053524]
  This paper presents a novel framework for modeling and conditional generation of 3D articulated objects.<n>We parameterize an articulated object as a tree of tokens and employ a transformer to generate both the object's high-level geometry code and its kinematic relations.<n>Our approach enables the generation of diverse objects with high-quality geometry and varying number of parts.
  arXiv  Detail & Related papers  (2024-12-10T07:00:05Z)
• MaterialPicker: Multi-Modal DiT-Based Material Generation [27.007661861644376]
  We propose a multi-modal material generator leveraging a Diffusion Transformer (DiT) architecture.<n>Our method can generate a material based on an image crop of a material sample, even if the captured surface is distorted.<n>We show that it enables more diverse material generation and better distortion correction than previous work.
  arXiv  Detail & Related papers  (2024-12-04T11:23:15Z)
• Edify 3D: Scalable High-Quality 3D Asset Generation [53.86838858460809]
  Edify 3D is an advanced solution designed for high-quality 3D asset generation. Our method can generate high-quality 3D assets with detailed geometry, clean shape topologies, high-resolution textures, and materials within 2 minutes of runtime.
  arXiv  Detail & Related papers  (2024-11-11T17:07:43Z)
• MaPa: Text-driven Photorealistic Material Painting for 3D Shapes [79.13775179541311]
  This paper aims to generate materials for 3D meshes from text descriptions.<n>Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs.<n>Our framework supports high-quality rendering and provides substantial flexibility in editing.
  arXiv  Detail & Related papers  (2024-04-26T17:54:38Z)
• Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials [108.59709545364395]
  GPT-4V can effectively recognize and describe materials, allowing the construction of a detailed material library. The correctly matched materials are then meticulously applied as reference for the new SVBRDF material generation. Make-it-Real offers a streamlined integration into the 3D content creation workflow.
  arXiv  Detail & Related papers  (2024-04-25T17:59:58Z)
• SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation [89.47132156950194]
  We present a novel framework built to simplify 3D asset generation for amateur users. Our method supports a variety of input modalities that can be easily provided by a human. Our model can combine all these tasks into one swiss-army-knife tool.
  arXiv  Detail & Related papers  (2022-12-08T18:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.