Text2Mesh: Text-Driven Neural Stylization for Meshes

• URL: http://arxiv.org/abs/2112.03221v1
• Date: Mon, 6 Dec 2021 18:23:29 GMT
• Title: Text2Mesh: Text-Driven Neural Stylization for Meshes
• Authors: Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, Rana Hanocka
• Abstract summary: Our framework, Text2Mesh, stylizes a 3D mesh by predicting color and local geometric details which conform to a target text prompt. We consider a disentangled representation of a 3D object using a fixed mesh input (content) coupled with a learned neural network, which we term neural style field network. In order to modify style, we obtain a similarity score between a text prompt (describing style) and a stylized mesh by harnessing the representational power of CLIP.
• Score: 18.435567297462416
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we develop intuitive controls for editing the style of 3D objects. Our framework, Text2Mesh, stylizes a 3D mesh by predicting color and local geometric details which conform to a target text prompt. We consider a disentangled representation of a 3D object using a fixed mesh input (content) coupled with a learned neural network, which we term neural style field network. In order to modify style, we obtain a similarity score between a text prompt (describing style) and a stylized mesh by harnessing the representational power of CLIP. Text2Mesh requires neither a pre-trained generative model nor a specialized 3D mesh dataset. It can handle low-quality meshes (non-manifold, boundaries, etc.) with arbitrary genus, and does not require UV parameterization. We demonstrate the ability of our technique to synthesize a myriad of styles over a wide variety of 3D meshes.

Related papers

• DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation [149.77077125310805]
  We present DreamMesh, a novel text-to-3D architecture that pivots on well-defined surfaces (triangle meshes) to generate high-fidelity explicit 3D model. In the coarse stage, the mesh is first deformed by text-guided Jacobians and then DreamMesh textures the mesh with an interlaced use of 2D diffusion models. In the fine stage, DreamMesh jointly manipulates the mesh and refines the texture map, leading to high-quality triangle meshes with high-fidelity textured materials.
  arXiv  Detail & Related papers  (2024-09-11T17:59:02Z)
• Bridging 3D Gaussian and Mesh for Freeview Video Rendering [57.21847030980905]
  GauMesh bridges the 3D Gaussian and Mesh for modeling and rendering the dynamic scenes. We show that our approach adapts the appropriate type of primitives to represent the different parts of the dynamic scene.
  arXiv  Detail & Related papers  (2024-03-18T04:01:26Z)
• 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models [102.75875255071246]
  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. We propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
  arXiv  Detail & Related papers  (2023-11-09T15:51:27Z)
• Mesh Neural Cellular Automata [62.101063045659906]
  We propose Mesh Neural Cellular Automata (MeshNCA), a method that directly synthesizes dynamic textures on 3D meshes without requiring any UV maps. Only trained on an Icosphere mesh, MeshNCA shows remarkable test-time generalization and can synthesize textures on unseen meshes in real time.
  arXiv  Detail & Related papers  (2023-11-06T01:54:37Z)
• Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models [21.622420436349245]
  We present Text2Room, a method for generating room-scale textured 3D meshes from a given text prompt as input. We leverage pre-trained 2D text-to-image models to synthesize a sequence of images from different poses. In order to lift these outputs into a consistent 3D scene representation, we combine monocular depth estimation with a text-conditioned inpainting model.
  arXiv  Detail & Related papers  (2023-03-21T16:21:02Z)
• 3D Highlighter: Localizing Regions on 3D Shapes via Text Descriptions [14.65300898522962]
  3D Highlighter is a technique for localizing semantic regions on a mesh using text as input. Our system demonstrates the ability to reason about where to place non-obviously related concepts on an input 3D shape.
  arXiv  Detail & Related papers  (2022-12-21T18:54:47Z)
• 3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations [81.45521258652734]
  We propose a method to create plausible geometric and texture style variations of 3D objects. Our method can create many novel stylized shapes, resulting in effortless 3D content creation and style-ware data augmentation.
  arXiv  Detail & Related papers  (2021-08-30T02:28:31Z)
• Deep Hybrid Self-Prior for Full 3D Mesh Generation [57.78562932397173]
  We propose to exploit a novel hybrid 2D-3D self-prior in deep neural networks to significantly improve the geometry quality. In particular, we first generate an initial mesh using a 3D convolutional neural network with 3D self-prior, and then encode both 3D information and color information in the 2D UV atlas. Our method recovers the 3D textured mesh model of high quality from sparse input, and outperforms the state-of-the-art methods in terms of both the geometry and texture quality.
  arXiv  Detail & Related papers  (2021-08-18T07:44:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.