ShaDDR: Interactive Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering

• URL: http://arxiv.org/abs/2306.04889v2
• Date: Wed, 22 Nov 2023 03:02:46 GMT
• Title: ShaDDR: Interactive Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering
• Authors: Qimin Chen, Zhiqin Chen, Hang Zhou, Hao Zhang
• Abstract summary: ShaDDR is an example-based deep generative neural network which produces a high-resolution textured 3D shape. Our method learns to detailize the geometry via multi-resolution voxel upsampling and generate textures on voxel surfaces. The generated shape preserves the overall structure of the input coarse voxel model.
• Score: 24.622120688131616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present ShaDDR, an example-based deep generative neural network which produces a high-resolution textured 3D shape through geometry detailization and conditional texture generation applied to an input coarse voxel shape. Trained on a small set of detailed and textured exemplar shapes, our method learns to detailize the geometry via multi-resolution voxel upsampling and generate textures on voxel surfaces via differentiable rendering against exemplar texture images from a few views. The generation is interactive, taking less than 1 second to produce a 3D model with voxel resolutions up to 512^3. The generated shape preserves the overall structure of the input coarse voxel model, while the style of the generated geometric details and textures can be manipulated through learned latent codes. In the experiments, we show that our method can generate higher-resolution shapes with plausible and improved geometric details and clean textures compared to prior works. Furthermore, we showcase the ability of our method to learn geometric details and textures from shapes reconstructed from real-world photos. In addition, we have developed an interactive modeling application to demonstrate the generalizability of our method to various user inputs and the controllability it offers, allowing users to interactively sculpt a coarse voxel shape to define the overall structure of the detailized 3D shape. Code and data are available at https://github.com/qiminchen/ShaDDR.

Related papers

• DetailGen3D: Generative 3D Geometry Enhancement via Data-Dependent Flow [44.72037991063735]
  DetailGen3D is a generative approach specifically designed to enhance generated 3D shapes. Our key insight is to model the coarse-to-fine transformation directly through data-dependent flows in latent space. We introduce a token matching strategy that ensures accurate spatial correspondence during refinement.
  arXiv  Detail & Related papers  (2024-11-25T17:08:17Z)
• DECOLLAGE: 3D Detailization by Controllable, Localized, and Learned Geometry Enhancement [38.719572669042925]
  We present a 3D modeling method which enables end-users to refine or detailize 3D shapes using machine learning. We show that our ability to localize details enables novel interactive creative and applications.
  arXiv  Detail & Related papers  (2024-09-10T00:51:49Z)
• CraftsMan: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refiner [34.78919665494048]
  CraftsMan can generate high-fidelity 3D geometries with highly varied shapes, regular mesh topologies, and detailed surfaces. Our method achieves high efficacy in producing superior-quality 3D assets compared to existing methods.
  arXiv  Detail & Related papers  (2024-05-23T18:30:12Z)
• ShapeClipper: Scalable 3D Shape Learning from Single-View Images via Geometric and CLIP-based Consistency [39.7058456335011]
  We present ShapeClipper, a novel method that reconstructs 3D object shapes from real-world single-view RGB images. ShapeClipper learns shape reconstruction from a set of single-view segmented images. We evaluate our method over three challenging real-world datasets.
  arXiv  Detail & Related papers  (2023-04-13T03:53:12Z)
• DreamStone: Image as Stepping Stone for Text-Guided 3D Shape Generation [105.97545053660619]
  We present a new text-guided 3D shape generation approach DreamStone. It uses images as a stepping stone to bridge the gap between text and shape modalities for generating 3D shapes without requiring paired text and 3D data. Our approach is generic, flexible, and scalable, and it can be easily integrated with various SVR models to expand the generative space and improve the generative fidelity.
  arXiv  Detail & Related papers  (2023-03-24T03:56:23Z)
• Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images [82.32776379815712]
  We study the problem of shape generation in 3D mesh representation from a small number of color images with or without camera poses. We adopt to further improve the shape quality by leveraging cross-view information with a graph convolution network. Our model is robust to the quality of the initial mesh and the error of camera pose, and can be combined with a differentiable function for test-time optimization.
  arXiv  Detail & Related papers  (2022-04-21T03:42:31Z)
• Facial Geometric Detail Recovery via Implicit Representation [147.07961322377685]
  We present a robust texture-guided geometric detail recovery approach using only a single in-the-wild facial image. Our method combines high-quality texture completion with the powerful expressiveness of implicit surfaces. Our method not only recovers accurate facial details but also decomposes normals, albedos, and shading parts in a self-supervised way.
  arXiv  Detail & Related papers  (2022-03-18T01:42:59Z)
• 3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations [81.45521258652734]
  We propose a method to create plausible geometric and texture style variations of 3D objects. Our method can create many novel stylized shapes, resulting in effortless 3D content creation and style-ware data augmentation.
  arXiv  Detail & Related papers  (2021-08-30T02:28:31Z)
• DECOR-GAN: 3D Shape Detailization by Conditional Refinement [50.8801457082181]
  We introduce a deep generative network for 3D shape detailization, akin to stylization with the style being geometric details. We demonstrate that our method can refine a coarse shape into a variety of detailed shapes with different styles.
  arXiv  Detail & Related papers  (2020-12-16T18:52:10Z)
• Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images [64.53227129573293]
  We investigate the problem of learning to generate 3D parametric surface representations for novel object instances, as seen from one or more views. We design neural networks capable of generating high-quality parametric 3D surfaces which are consistent between views. Our method is supervised and trained on a public dataset of shapes from common object categories.
  arXiv  Detail & Related papers  (2020-08-18T06:33:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.