Capturing and Animation of Body and Clothing from Monocular Video

• URL: http://arxiv.org/abs/2210.01868v1
• Date: Tue, 4 Oct 2022 19:34:05 GMT
• Title: Capturing and Animation of Body and Clothing from Monocular Video
• Authors: Yao Feng, Jinlong Yang, Marc Pollefeys, Michael J. Black, Timo Bolkart
• Abstract summary: We present SCARF, a hybrid model combining a mesh-based body with a neural radiance field. integrating the mesh into the rendering enables us to optimize SCARF directly from monocular videos. We demonstrate that SCARFs clothing with higher visual quality than existing methods, that the clothing deforms with changing body pose and body shape, and that clothing can be successfully transferred between avatars of different subjects.
• Score: 105.87228128022804
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While recent work has shown progress on extracting clothed 3D human avatars from a single image, video, or a set of 3D scans, several limitations remain. Most methods use a holistic representation to jointly model the body and clothing, which means that the clothing and body cannot be separated for applications like virtual try-on. Other methods separately model the body and clothing, but they require training from a large set of 3D clothed human meshes obtained from 3D/4D scanners or physics simulations. Our insight is that the body and clothing have different modeling requirements. While the body is well represented by a mesh-based parametric 3D model, implicit representations and neural radiance fields are better suited to capturing the large variety in shape and appearance present in clothing. Building on this insight, we propose SCARF (Segmented Clothed Avatar Radiance Field), a hybrid model combining a mesh-based body with a neural radiance field. Integrating the mesh into the volumetric rendering in combination with a differentiable rasterizer enables us to optimize SCARF directly from monocular videos, without any 3D supervision. The hybrid modeling enables SCARF to (i) animate the clothed body avatar by changing body poses (including hand articulation and facial expressions), (ii) synthesize novel views of the avatar, and (iii) transfer clothing between avatars in virtual try-on applications. We demonstrate that SCARF reconstructs clothing with higher visual quality than existing methods, that the clothing deforms with changing body pose and body shape, and that clothing can be successfully transferred between avatars of different subjects. The code and models are available at https://github.com/YadiraF/SCARF.

Related papers

• Synthesizing Moving People with 3D Control [88.68284137105654]
  We present a diffusion model-based framework for animating people from a single image for a given target 3D motion sequence. For the first part, we learn an in-filling diffusion model to hallucinate unseen parts of a person given a single image. Second, we develop a diffusion-based rendering pipeline, which is controlled by 3D human poses.
  arXiv  Detail & Related papers  (2024-01-19T18:59:11Z)
• Learning Disentangled Avatars with Hybrid 3D Representations [102.9632315060652]
  We present Disentangled Avatars(DELTA) which models humans with hybrid explicit-implicit 3D representations. We consider the disentanglement of the human body and clothing and in the second, we disentangle the face and hair. We show how these two applications can be easily combined to model full-body avatars.
  arXiv  Detail & Related papers  (2023-09-12T17:59:36Z)
• AvatarFusion: Zero-shot Generation of Clothing-Decoupled 3D Avatars Using 2D Diffusion [34.609403685504944]
  We present AvatarFusion, a framework for zero-shot text-to-avatar generation. We use a latent diffusion model to provide pixel-level guidance for generating human-realistic avatars. We also introduce a novel optimization method, called Pixel-Semantics Difference-Sampling (PS-DS), which semantically separates the generation of body and clothes.
  arXiv  Detail & Related papers  (2023-07-13T02:19:56Z)
• Realistic, Animatable Human Reconstructions for Virtual Fit-On [0.7649716717097428]
  We present an end-to-end virtual try-on pipeline, that can fit different clothes on a personalized 3-D human model. Our main idea is to construct an animatable 3-D human model and try-on different clothes in a 3-D virtual environment.
  arXiv  Detail & Related papers  (2022-10-16T13:36:24Z)
• gDNA: Towards Generative Detailed Neural Avatars [94.9804106939663]
  We show that our model is able to generate natural human avatars wearing diverse and detailed clothing. Our method can be used on the task of fitting human models to raw scans, outperforming the previous state-of-the-art.
  arXiv  Detail & Related papers  (2022-01-11T18:46:38Z)
• The Power of Points for Modeling Humans in Clothing [60.00557674969284]
  Currently it requires an artist to create 3D human avatars with realistic clothing that can move naturally. We show that a 3D representation can capture varied topology at high resolution and that can be learned from data. We train a neural network with a novel local clothing geometric feature to represent the shape of different outfits.
  arXiv  Detail & Related papers  (2021-09-02T17:58:45Z)
• Explicit Clothing Modeling for an Animatable Full-Body Avatar [21.451440299450592]
  We build an animatable clothed body avatar with an explicit representation of the clothing on the upper body from multi-view captured videos. To learn the interaction between the body dynamics and clothing states, we use a temporal convolution network to predict the clothing latent code. We show photorealistic animation output for three different actors, and demonstrate the advantage of our clothed-body avatars over single-layer avatars.
  arXiv  Detail & Related papers  (2021-06-28T17:58:40Z)
• Neural 3D Clothes Retargeting from a Single Image [91.5030622330039]
  We present a method of clothes; generating the potential poses and deformations of a given 3D clothing template model to fit onto a person in a single RGB image. The problem is fundamentally ill-posed as attaining the ground truth data is impossible, i.e. images of people wearing the different 3D clothing template model model at exact same pose. We propose a semi-supervised learning framework that validates the physical plausibility of 3D deformation by matching with the prescribed body-to-cloth contact points and clothing to fit onto the unlabeled silhouette.
  arXiv  Detail & Related papers  (2021-01-29T20:50:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.