Single Mesh Diffusion Models with Field Latents for Texture Generation

• URL: http://arxiv.org/abs/2312.09250v3
• Date: Tue, 28 May 2024 20:05:49 GMT
• Title: Single Mesh Diffusion Models with Field Latents for Texture Generation
• Authors: Thomas W. Mitchel, Carlos Esteves, Ameesh Makadia,
• Abstract summary: We introduce a framework for intrinsic latent diffusion models operating directly on the surfaces of 3D shapes. We consider a single-textured-mesh paradigm, where our models are trained to generate variations of a given texture on a mesh. Our models can also be adapted for user-controlled editing tasks such as inpainting and label-guided generation.
• Score: 18.78126579775479
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a framework for intrinsic latent diffusion models operating directly on the surfaces of 3D shapes, with the goal of synthesizing high-quality textures. Our approach is underpinned by two contributions: field latents, a latent representation encoding textures as discrete vector fields on the mesh vertices, and field latent diffusion models, which learn to denoise a diffusion process in the learned latent space on the surface. We consider a single-textured-mesh paradigm, where our models are trained to generate variations of a given texture on a mesh. We show the synthesized textures are of superior fidelity compared those from existing single-textured-mesh generative models. Our models can also be adapted for user-controlled editing tasks such as inpainting and label-guided generation. The efficacy of our approach is due in part to the equivariance of our proposed framework under isometries, allowing our models to seamlessly reproduce details across locally similar regions and opening the door to a notion of generative texture transfer.

Related papers

• DreamPolish: Domain Score Distillation With Progressive Geometry Generation [66.94803919328815]
  We introduce DreamPolish, a text-to-3D generation model that excels in producing refined geometry and high-quality textures. In the geometry construction phase, our approach leverages multiple neural representations to enhance the stability of the synthesis process. In the texture generation phase, we introduce a novel score distillation objective, namely domain score distillation (DSD), to guide neural representations toward such a domain.
  arXiv  Detail & Related papers  (2024-11-03T15:15:01Z)
• SpaceMesh: A Continuous Representation for Learning Manifold Surface Meshes [61.110517195874074]
  We present a scheme to directly generate manifold, polygonal meshes of complex connectivity as the output of a neural network. Our key innovation is to define a continuous latent connectivity space at each mesh, which implies the discrete mesh. In applications, this approach not only yields high-quality outputs from generative models, but also enables directly learning challenging geometry processing tasks such as mesh repair.
  arXiv  Detail & Related papers  (2024-09-30T17:59:03Z)
• Infinite Texture: Text-guided High Resolution Diffusion Texture Synthesis [61.189479577198846]
  We present Infinite Texture, a method for generating arbitrarily large texture images from a text prompt. Our approach fine-tunes a diffusion model on a single texture, and learns to embed that statistical distribution in the output domain of the model. At generation time, our fine-tuned diffusion model is used through a score aggregation strategy to generate output texture images of arbitrary resolution on a single GPU.
  arXiv  Detail & Related papers  (2024-05-13T21:53:09Z)
• Semantic Human Mesh Reconstruction with Textures [40.613861470128256]
  SHERT is a novel pipeline that reconstructs semantic human meshes with textures and high-precision details. Our reconstructed meshes have stable UV unwrapping, high-quality triangle meshes, and consistent semantic information.
  arXiv  Detail & Related papers  (2024-03-05T00:34:05Z)
• Binary Opacity Grids: Capturing Fine Geometric Detail for Mesh-Based View Synthesis [70.40950409274312]
  We modify density fields to encourage them to converge towards surfaces, without compromising their ability to reconstruct thin structures. We also develop a fusion-based meshing strategy followed by mesh simplification and appearance model fitting. The compact meshes produced by our model can be rendered in real-time on mobile devices.
  arXiv  Detail & Related papers  (2024-02-19T18:59:41Z)
• Diffusion-based Holistic Texture Rectification and Synthesis [26.144666226217062]
  Traditional texture synthesis approaches focus on generating textures from pristine samples. We propose a framework that synthesizes holistic textures from degraded samples in natural images.
  arXiv  Detail & Related papers  (2023-09-26T08:44:46Z)
• Controllable Mesh Generation Through Sparse Latent Point Diffusion Models [105.83595545314334]
  We design a novel sparse latent point diffusion model for mesh generation. Our key insight is to regard point clouds as an intermediate representation of meshes, and model the distribution of point clouds instead. Our proposed sparse latent point diffusion model achieves superior performance in terms of generation quality and controllability.
  arXiv  Detail & Related papers  (2023-03-14T14:25:29Z)
• Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement [78.48648360358193]
  We present a novel framework that generates textured surface meshes from images. Our approach begins by efficiently initializing the geometry and view-dependency appearance with a NeRF. We jointly refine the appearance with geometry and bake it into texture images for real-time rendering.
  arXiv  Detail & Related papers  (2023-03-03T17:14:44Z)
• Vehicle Reconstruction and Texture Estimation Using Deep Implicit Semantic Template Mapping [32.580904361799966]
  We introduce VERTEX, an effective solution to recover 3D shape and intrinsic texture of vehicles from uncalibrated monocular input. By fusing the global and local features together, our approach is capable to generate consistent and detailed texture in both visible and invisible areas.
  arXiv  Detail & Related papers  (2020-11-30T09:27:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.