Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials

• URL: http://arxiv.org/abs/2407.02445v1
• Date: Tue, 2 Jul 2024 17:21:47 GMT
• Title: Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials
• Authors: Yawar Siddiqui, Tom Monnier, Filippos Kokkinos, Mahendra Kariya, Yanir Kleiman, Emilien Garreau, Oran Gafni, Natalia Neverova, Andrea Vedaldi, Roman Shapovalov, David Novotny,
• Abstract summary: AssetGen is a significant advancement in text-to-3D generation. It produces faithful, high-quality meshes with texture and material control.
• Score: 58.178540282148475
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Meta 3D AssetGen (AssetGen), a significant advancement in text-to-3D generation which produces faithful, high-quality meshes with texture and material control. Compared to works that bake shading in the 3D object's appearance, AssetGen outputs physically-based rendering (PBR) materials, supporting realistic relighting. AssetGen generates first several views of the object with factored shaded and albedo appearance channels, and then reconstructs colours, metalness and roughness in 3D, using a deferred shading loss for efficient supervision. It also uses a sign-distance function to represent 3D shape more reliably and introduces a corresponding loss for direct shape supervision. This is implemented using fused kernels for high memory efficiency. After mesh extraction, a texture refinement transformer operating in UV space significantly improves sharpness and details. AssetGen achieves 17% improvement in Chamfer Distance and 40% in LPIPS over the best concurrent work for few-view reconstruction, and a human preference of 72% over the best industry competitors of comparable speed, including those that support PBR. Project page with generated assets: https://assetgen.github.io

Related papers

• Boosting 3D Object Generation through PBR Materials [32.732511476490316]
  We propose a novel approach to boost the quality of generated 3D objects from the perspective of Physics-Based Rendering (PBR) materials. For albedo and bump maps, we leverage Stable Diffusion fine-tuned on synthetic data to extract these values. In terms of roughness and metalness maps, we adopt a semi-automatic process to provide room for interactive adjustment.
  arXiv  Detail & Related papers  (2024-11-25T04:20:52Z)
• Meta 3D Gen [57.313835190702484]
  3DGen offers 3D asset creation with high prompt fidelity and high-quality 3D shapes and textures in under a minute. 3DGen supports physically-based rendering (PBR), necessary for 3D asset relighting in real-world applications.
  arXiv  Detail & Related papers  (2024-07-02T18:37:52Z)
• GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality [99.63429416013713]
  3D-GS has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. We propose a novel framework named GaussianDreamerPro to enhance the generation quality.
  arXiv  Detail & Related papers  (2024-06-26T16:12:09Z)
• MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformers [76.70891862458384]
  We introduce MeshAnything, a model that treats mesh extraction as a generation problem. By converting 3D assets in any 3D representation into AMs, MeshAnything can be integrated with various 3D asset production methods. Our method generates AMs with hundreds of times fewer faces, significantly improving storage, rendering, and simulation efficiencies.
  arXiv  Detail & Related papers  (2024-06-14T16:30:25Z)
• Texture-GS: Disentangling the Geometry and Texture for 3D Gaussian Splatting Editing [79.10630153776759]
  3D Gaussian splatting, emerging as a groundbreaking approach, has drawn increasing attention for its capabilities of high-fidelity reconstruction and real-time rendering. We propose a novel approach, namely Texture-GS, to disentangle the appearance from the geometry by representing it as a 2D texture mapped onto the 3D surface. Our method not only facilitates high-fidelity appearance editing but also achieves real-time rendering on consumer-level devices.
  arXiv  Detail & Related papers  (2024-03-15T06:42:55Z)
• IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation [96.32684334038278]
  In this paper, we explore the design space of text-to-3D models. We significantly improve multi-view generation by considering video instead of image generators. Our new method, IM-3D, reduces the number of evaluations of the 2D generator network 10-100x.
  arXiv  Detail & Related papers  (2024-02-13T18:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.