SMPL-GPTexture: Dual-View 3D Human Texture Estimation using Text-to-Image Generation Models

• URL: http://arxiv.org/abs/2504.13378v1
• Date: Thu, 17 Apr 2025 23:28:38 GMT
• Title: SMPL-GPTexture: Dual-View 3D Human Texture Estimation using Text-to-Image Generation Models
• Authors: Mingxiao Tu, Shuchang Ye, Hoijoon Jung, Jinman Kim,
• Abstract summary: SMPL-GPTexture is a novel pipeline that takes natural language prompts as input and leverages a state-of-the-art text-to-image generation model.<n>We show that our pipeline can generate high resolution texture aligned with user's prompts.
• Score: 7.436391283592317
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating high-quality, photorealistic textures for 3D human avatars remains a fundamental yet challenging task in computer vision and multimedia field. However, real paired front and back images of human subjects are rarely available with privacy, ethical and cost of acquisition, which restricts scalability of the data. Additionally, learning priors from image inputs using deep generative models, such as GANs or diffusion models, to infer unseen regions such as the human back often leads to artifacts, structural inconsistencies, or loss of fine-grained detail. To address these issues, we present SMPL-GPTexture (skinned multi-person linear model - general purpose Texture), a novel pipeline that takes natural language prompts as input and leverages a state-of-the-art text-to-image generation model to produce paired high-resolution front and back images of a human subject as the starting point for texture estimation. Using the generated paired dual-view images, we first employ a human mesh recovery model to obtain a robust 2D-to-3D SMPL alignment between image pixels and the 3D model's UV coordinates for each views. Second, we use an inverted rasterization technique that explicitly projects the observed colour from the input images into the UV space, thereby producing accurate, complete texture maps. Finally, we apply a diffusion-based inpainting module to fill in the missing regions, and the fusion mechanism then combines these results into a unified full texture map. Extensive experiments shows that our SMPL-GPTexture can generate high resolution texture aligned with user's prompts.

Related papers

• RomanTex: Decoupling 3D-aware Rotary Positional Embedded Multi-Attention Network for Texture Synthesis [10.350576861948952]
  RomanTex is a multiview-based texture generation framework that integrates a multi-attention network with an underlying 3D representation.<n>Our method achieves state-of-the-art results in texture quality and consistency.
  arXiv  Detail & Related papers  (2025-03-24T17:56:11Z)
• TEXGen: a Generative Diffusion Model for Mesh Textures [63.43159148394021]
  We focus on the fundamental problem of learning in the UV texture space itself. We propose a scalable network architecture that interleaves convolutions on UV maps with attention layers on point clouds. We train a 700 million parameter diffusion model that can generate UV texture maps guided by text prompts and single-view images.
  arXiv  Detail & Related papers  (2024-11-22T05:22:11Z)
• FAMOUS: High-Fidelity Monocular 3D Human Digitization Using View Synthesis [51.193297565630886]
  The challenge of accurately inferring texture remains, particularly in obscured areas such as the back of a person in frontal-view images. This limitation in texture prediction largely stems from the scarcity of large-scale and diverse 3D datasets. We propose leveraging extensive 2D fashion datasets to enhance both texture and shape prediction in 3D human digitization.
  arXiv  Detail & Related papers  (2024-10-13T01:25:05Z)
• TexDreamer: Towards Zero-Shot High-Fidelity 3D Human Texture Generation [41.959089177835764]
  TexDreamer is the first zero-shot multimodal high-fidelity 3D human texture generation model. We introduce ArTicuLated humAn textureS, the largest high-resolution (1024 X 1024) 3D human texture dataset.
  arXiv  Detail & Related papers  (2024-03-19T17:02:07Z)
• UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling [71.87807614875497]
  We propose UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures. We collect and process a new dataset of human motion, which includes multi-view images, scanned models, parametric model registration, and corresponding texture maps. Experimental results demonstrate that our method achieves state-of-the-art synthesis of novel view and novel pose.
  arXiv  Detail & Related papers  (2024-03-18T09:03:56Z)
• ConTex-Human: Free-View Rendering of Human from a Single Image with Texture-Consistent Synthesis [49.28239918969784]
  We introduce a texture-consistent back view synthesis module that could transfer the reference image content to the back view. We also propose a visibility-aware patch consistency regularization for texture mapping and refinement combined with the synthesized back view texture.
  arXiv  Detail & Related papers  (2023-11-28T13:55:53Z)
• GETAvatar: Generative Textured Meshes for Animatable Human Avatars [69.56959932421057]
  We study the problem of 3D-aware full-body human generation, aiming at creating animatable human avatars with high-quality geometries and textures. We propose GETAvatar, a Generative model that directly generates Explicit Textured 3D rendering for animatable human Avatar.
  arXiv  Detail & Related papers  (2023-10-04T10:30:24Z)
• OSTeC: One-Shot Texture Completion [86.23018402732748]
  We propose an unsupervised approach for one-shot 3D facial texture completion. The proposed approach rotates an input image in 3D and fill-in the unseen regions by reconstructing the rotated image in a 2D face generator. We frontalize the target image by projecting the completed texture into the generator.
  arXiv  Detail & Related papers  (2020-12-30T23:53:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.