UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling

• URL: http://arxiv.org/abs/2403.11589v1
• Date: Mon, 18 Mar 2024 09:03:56 GMT
• Title: UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling
• Authors: Yujiao Jiang, Qingmin Liao, Xiaoyu Li, Li Ma, Qi Zhang, Chaopeng Zhang, Zongqing Lu, Ying Shan,
• Abstract summary: We propose UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures. We collect and process a new dataset of human motion, which includes multi-view images, scanned models, parametric model registration, and corresponding texture maps. Experimental results demonstrate that our method achieves state-of-the-art synthesis of novel view and novel pose.
• Score: 71.87807614875497
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing photo-realistic drivable human avatars from multi-view image sequences has been a popular and challenging topic in the field of computer vision and graphics. While existing NeRF-based methods can achieve high-quality novel view rendering of human models, both training and inference processes are time-consuming. Recent approaches have utilized 3D Gaussians to represent the human body, enabling faster training and rendering. However, they undermine the importance of the mesh guidance and directly predict Gaussians in 3D space with coarse mesh guidance. This hinders the learning procedure of the Gaussians and tends to produce blurry textures. Therefore, we propose UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures. We utilize the embedding of UV map to learn Gaussian textures in 2D space, leveraging the capabilities of powerful 2D networks to extract features. Additionally, through an independent Mesh network, we optimize pose-dependent geometric deformations, thereby guiding Gaussian rendering and significantly enhancing rendering quality. We collect and process a new dataset of human motion, which includes multi-view images, scanned models, parametric model registration, and corresponding texture maps. Experimental results demonstrate that our method achieves state-of-the-art synthesis of novel view and novel pose. The code and data will be made available on the homepage https://alex-jyj.github.io/UV-Gaussians/ once the paper is accepted.

Related papers

• FAMOUS: High-Fidelity Monocular 3D Human Digitization Using View Synthesis [51.193297565630886]
  The challenge of accurately inferring texture remains, particularly in obscured areas such as the back of a person in frontal-view images. This limitation in texture prediction largely stems from the scarcity of large-scale and diverse 3D datasets. We propose leveraging extensive 2D fashion datasets to enhance both texture and shape prediction in 3D human digitization.
  arXiv  Detail & Related papers  (2024-10-13T01:25:05Z)
• DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation [10.250715657201363]
  We introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video. Our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.
  arXiv  Detail & Related papers  (2024-10-09T10:41:08Z)
• UVMap-ID: A Controllable and Personalized UV Map Generative Model [67.71022515856653]
  We introduce UVMap-ID, a controllable and personalized UV Map generative model. Unlike traditional large-scale training methods in 2D, we propose to fine-tune a pre-trained text-to-image diffusion model. Both quantitative and qualitative analyses demonstrate the effectiveness of our method in controllable and personalized UV Map generation.
  arXiv  Detail & Related papers  (2024-04-22T20:30:45Z)
• Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
  We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence. ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images. Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
  arXiv  Detail & Related papers  (2023-12-22T20:56:46Z)
• Animatable 3D Gaussians for High-fidelity Synthesis of Human Motions [37.50707388577952]
  We present a novel animatable 3D Gaussian model for rendering high-fidelity free-view human motions in real time. Compared to existing NeRF-based methods, the model owns better capability in high-frequency details without the jittering problem across video frames.
  arXiv  Detail & Related papers  (2023-11-22T14:00:23Z)
• GETAvatar: Generative Textured Meshes for Animatable Human Avatars [69.56959932421057]
  We study the problem of 3D-aware full-body human generation, aiming at creating animatable human avatars with high-quality geometries and textures. We propose GETAvatar, a Generative model that directly generates Explicit Textured 3D rendering for animatable human Avatar.
  arXiv  Detail & Related papers  (2023-10-04T10:30:24Z)
• VeRi3D: Generative Vertex-based Radiance Fields for 3D Controllable Human Image Synthesis [27.81573705217842]
  We propose VeRi3D, a generative human radiance field parameterized by vertices of the parametric human template, SMPL. We demonstrate that our simple approach allows for generating photorealistic human images with free control over camera pose, human pose, shape, as well as enabling part-level editing.
  arXiv  Detail & Related papers  (2023-09-09T13:53:29Z)
• DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars [92.37436369781692]
  We present DRaCoN, a framework for learning full-body volumetric avatars. It exploits the advantages of both the 2D and 3D neural rendering techniques. Experiments on the challenging ZJU-MoCap and Human3.6M datasets indicate that DRaCoN outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T17:59:15Z)
• Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction [0.0]
  Learning-based approaches for 3D model reconstruction have attracted attention owing to its modern applications. We present a novel sampling algorithm by optimizing the gradient of predicted coordinates based on the variance on the sampling image. We also adopt Frechet Inception Distance (FID) to form a loss function in learning, which helps bridging the gap between rendered images and input images.
  arXiv  Detail & Related papers  (2021-04-29T07:52:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.