Semantic Human Mesh Reconstruction with Textures

• URL: http://arxiv.org/abs/2403.02561v2
• Date: Wed, 3 Apr 2024 05:43:10 GMT
• Title: Semantic Human Mesh Reconstruction with Textures
• Authors: Xiaoyu Zhan, Jianxin Yang, Yuanqi Li, Jie Guo, Yanwen Guo, Wenping Wang,
• Abstract summary: SHERT is a novel pipeline that reconstructs semantic human meshes with textures and high-precision details. Our reconstructed meshes have stable UV unwrapping, high-quality triangle meshes, and consistent semantic information.
• Score: 40.613861470128256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The field of 3D detailed human mesh reconstruction has made significant progress in recent years. However, current methods still face challenges when used in industrial applications due to unstable results, low-quality meshes, and a lack of UV unwrapping and skinning weights. In this paper, we present SHERT, a novel pipeline that can reconstruct semantic human meshes with textures and high-precision details. SHERT applies semantic- and normal-based sampling between the detailed surface (e.g. mesh and SDF) and the corresponding SMPL-X model to obtain a partially sampled semantic mesh and then generates the complete semantic mesh by our specifically designed self-supervised completion and refinement networks. Using the complete semantic mesh as a basis, we employ a texture diffusion model to create human textures that are driven by both images and texts. Our reconstructed meshes have stable UV unwrapping, high-quality triangle meshes, and consistent semantic information. The given SMPL-X model provides semantic information and shape priors, allowing SHERT to perform well even with incorrect and incomplete inputs. The semantic information also makes it easy to substitute and animate different body parts such as the face, body, and hands. Quantitative and qualitative experiments demonstrate that SHERT is capable of producing high-fidelity and robust semantic meshes that outperform state-of-the-art methods.

Related papers

• DreamPolish: Domain Score Distillation With Progressive Geometry Generation [66.94803919328815]
  We introduce DreamPolish, a text-to-3D generation model that excels in producing refined geometry and high-quality textures. In the geometry construction phase, our approach leverages multiple neural representations to enhance the stability of the synthesis process. In the texture generation phase, we introduce a novel score distillation objective, namely domain score distillation (DSD), to guide neural representations toward such a domain.
  arXiv  Detail & Related papers  (2024-11-03T15:15:01Z)
• Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering [11.228453237603834]
  We present a novel fine-grained multi-view hand mesh reconstruction method that leverages inverse rendering to restore hand poses and intricate details. We also introduce a novel Hand Albedo and Mesh (HAM) optimization module to refine both the hand mesh and textures. Our proposed approach outperforms the state-of-the-art methods on both reconstruction accuracy and rendering quality.
  arXiv  Detail & Related papers  (2024-07-08T07:28:24Z)
• HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions [68.28684509445529]
  We present HandBooster, a new approach to uplift the data diversity and boost the 3D hand-mesh reconstruction performance. First, we construct versatile content-aware conditions to guide a diffusion model to produce realistic images with diverse hand appearances, poses, views, and backgrounds. Then, we design a novel condition creator based on our similarity-aware distribution sampling strategies to deliberately find novel and realistic interaction poses that are distinctive from the training set.
  arXiv  Detail & Related papers  (2024-03-27T13:56:08Z)
• Single Mesh Diffusion Models with Field Latents for Texture Generation [18.78126579775479]
  We introduce a framework for intrinsic latent diffusion models operating directly on the surfaces of 3D shapes. We consider a single-textured-mesh paradigm, where our models are trained to generate variations of a given texture on a mesh. Our models can also be adapted for user-controlled editing tasks such as inpainting and label-guided generation.
  arXiv  Detail & Related papers  (2023-12-14T18:59:36Z)
• PaintHuman: Towards High-fidelity Text-to-3D Human Texturing via Denoised Score Distillation [89.09455618184239]
  Recent advances in text-to-3D human generation have been groundbreaking. We propose a model called PaintHuman to address the challenges from two aspects. We use the depth map as a guidance to ensure realistic semantically aligned textures.
  arXiv  Detail & Related papers  (2023-10-14T00:37:16Z)
• HiFiHR: Enhancing 3D Hand Reconstruction from a Single Image via High-Fidelity Texture [40.012406098563204]
  We present HiFiHR, a high-fidelity hand reconstruction approach that utilizes render-and-compare in the learning-based framework from a single image. Experimental results on public benchmarks including FreiHAND and HO-3D demonstrate that our method outperforms the state-of-the-art hand reconstruction methods in texture reconstruction quality.
  arXiv  Detail & Related papers  (2023-08-25T18:48:40Z)
• Mesh Draping: Parametrization-Free Neural Mesh Transfer [92.55503085245304]
  Mesh Draping is a neural method for transferring existing mesh structure from one shape to another. We show that by leveraging gradually increasing frequencies to guide the neural optimization, we are able to achieve stable and high quality mesh transfer.
  arXiv  Detail & Related papers  (2021-10-11T17:24:52Z)
• Topologically Consistent Multi-View Face Inference Using Volumetric Sampling [25.001398662643986]
  ToFu is a geometry inference framework that can produce topologically consistent meshes across identities and expressions. A novel progressive mesh generation network embeds the topological structure of the face in a feature volume. These high-quality assets are readily usable by production studios for avatar creation, animation and physically-based skin rendering.
  arXiv  Detail & Related papers  (2021-10-06T17:55:08Z)
• Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation [75.59415852802958]
  Shape-My-Face (SMF) is a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model. Our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets.
  arXiv  Detail & Related papers  (2020-12-16T20:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.