TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition

• URL: http://arxiv.org/abs/2210.11277v1
• Date: Thu, 20 Oct 2022 13:52:18 GMT
• Title: TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition
• Authors: Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, Kui Jia
• Abstract summary: We propose TANGO, which transfers the appearance style of a given 3D shape according to a text prompt in a photorealistic manner. We show that TANGO outperforms existing methods of text-driven 3D style transfer in terms of photorealistic quality, consistency of 3D geometry, and robustness when stylizing low-quality meshes.
• Score: 39.312567993736025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Creation of 3D content by stylization is a promising yet challenging problem in computer vision and graphics research. In this work, we focus on stylizing photorealistic appearance renderings of a given surface mesh of arbitrary topology. Motivated by the recent surge of cross-modal supervision of the Contrastive Language-Image Pre-training (CLIP) model, we propose TANGO, which transfers the appearance style of a given 3D shape according to a text prompt in a photorealistic manner. Technically, we propose to disentangle the appearance style as the spatially varying bidirectional reflectance distribution function, the local geometric variation, and the lighting condition, which are jointly optimized, via supervision of the CLIP loss, by a spherical Gaussians based differentiable renderer. As such, TANGO enables photorealistic 3D style transfer by automatically predicting reflectance effects even for bare, low-quality meshes, without training on a task-specific dataset. Extensive experiments show that TANGO outperforms existing methods of text-driven 3D style transfer in terms of photorealistic quality, consistency of 3D geometry, and robustness when stylizing low-quality meshes. Our codes and results are available at our project webpage https://cyw-3d.github.io/tango/.

Related papers

• 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models [102.75875255071246]
  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. We propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
  arXiv  Detail & Related papers  (2023-11-09T15:51:27Z)
• Directional Texture Editing for 3D Models [51.31499400557996]
  ITEM3D is designed for automatic textbf3D object editing according to the text textbfInstructions. Leveraging the diffusion models and the differentiable rendering, ITEM3D takes the rendered images as the bridge of text and 3D representation.
  arXiv  Detail & Related papers  (2023-09-26T12:01:13Z)
• Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization [91.52882218901627]
  We propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing. Our method improves upon photo-realism, geometry, and expression accuracy compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T17:58:40Z)
• Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation [45.69270771487455]
  We propose a new method of Fantasia3D for high-quality text-to-3D content creation. Key to Fantasia3D is the disentangled modeling and learning of geometry and appearance. Our framework is more compatible with popular graphics engines, supporting relighting, editing, and physical simulation of the generated 3D assets.
  arXiv  Detail & Related papers  (2023-03-24T09:30:09Z)
• Texturify: Generating Textures on 3D Shape Surfaces [34.726179801982646]
  We propose Texturify to learn a 3D shape that predicts texture on the 3D input. Our method does not require any 3D color supervision to learn 3D objects.
  arXiv  Detail & Related papers  (2022-04-05T18:00:04Z)
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis [163.96778522283967]
  We propose a shading-guided generative implicit model that is able to learn a starkly improved shape representation. An accurate 3D shape should also yield a realistic rendering under different lighting conditions. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2021-10-29T10:53:12Z)
• Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
  We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
  arXiv  Detail & Related papers  (2021-09-13T15:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.