Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Video

• URL: http://arxiv.org/abs/2211.12782v2
• Date: Wed, 22 Mar 2023 09:08:09 GMT
• Title: Hand Avatar: Free-Pose Hand Animation and Rendering from Monocular Video
• Authors: Xingyu Chen, Baoyuan Wang, Heung-Yeung Shum
• Abstract summary: We present HandAvatar, a novel representation for hand animation and rendering. HandAvatar can generate smoothly compositional geometry and self-occlusion-aware texture.
• Score: 23.148367696192107
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present HandAvatar, a novel representation for hand animation and rendering, which can generate smoothly compositional geometry and self-occlusion-aware texture. Specifically, we first develop a MANO-HD model as a high-resolution mesh topology to fit personalized hand shapes. Sequentially, we decompose hand geometry into per-bone rigid parts, and then re-compose paired geometry encodings to derive an across-part consistent occupancy field. As for texture modeling, we propose a self-occlusion-aware shading field (SelF). In SelF, drivable anchors are paved on the MANO-HD surface to record albedo information under a wide variety of hand poses. Moreover, directed soft occupancy is designed to describe the ray-to-surface relation, which is leveraged to generate an illumination field for the disentanglement of pose-independent albedo and pose-dependent illumination. Trained from monocular video data, our HandAvatar can perform free-pose hand animation and rendering while at the same time achieving superior appearance fidelity. We also demonstrate that HandAvatar provides a route for hand appearance editing. Project website: https://seanchenxy.github.io/HandAvatarWeb.

Related papers

• XHand: Real-time Expressive Hand Avatar [9.876680405587745]
  We introduce an expressive hand avatar, named XHand, that is designed to generate hand shape, appearance, and deformations in real-time. XHand is able to recover high-fidelity geometry and texture for hand animations across diverse poses in real-time.
  arXiv  Detail & Related papers  (2024-07-30T17:49:21Z)
• VividPose: Advancing Stable Video Diffusion for Realistic Human Image Animation [79.99551055245071]
  We propose VividPose, an end-to-end pipeline that ensures superior temporal stability. An identity-aware appearance controller integrates additional facial information without compromising other appearance details. A geometry-aware pose controller utilizes both dense rendering maps from SMPL-X and sparse skeleton maps. VividPose exhibits superior generalization capabilities on our proposed in-the-wild dataset.
  arXiv  Detail & Related papers  (2024-05-28T13:18:32Z)
• TexVocab: Texture Vocabulary-conditioned Human Avatars [42.170169762733835]
  TexVocab is a novel avatar representation that constructs a texture vocabulary and associates body poses with texture maps for animation. Our method is able to create animatable human avatars with detailed and dynamic appearances from RGB videos.
  arXiv  Detail & Related papers  (2024-03-31T01:58:04Z)
• Democratizing the Creation of Animatable Facial Avatars [2.1740466069378597]
  We propose a novel pipeline for obtaining geometry and texture without using a light stage or any other high-end hardware. A key novel idea consists of warping real-world images to align with the geometry of a template avatar. Not only can our method be used to obtain a neutral expression geometry and de-lit texture, but it can also be used to improve avatars after they have been imported into an animation system.
  arXiv  Detail & Related papers  (2024-01-29T20:14:40Z)
• FLARE: Fast Learning of Animatable and Relightable Mesh Avatars [64.48254296523977]
  Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. We introduce FLARE, a technique that enables the creation of animatable and relightable avatars from a single monocular video.
  arXiv  Detail & Related papers  (2023-10-26T16:13:00Z)
• TADA! Text to Animatable Digital Avatars [57.52707683788961]
  TADA takes textual descriptions and produces expressive 3D avatars with high-quality geometry and lifelike textures. We derive an optimizable high-resolution body model from SMPL-X with 3D displacements and a texture map. We render normals and RGB images of the generated character and exploit their latent embeddings in the SDS training process.
  arXiv  Detail & Related papers  (2023-08-21T17:59:10Z)
• AvatarReX: Real-time Expressive Full-body Avatars [35.09470037950997]
  We present AvatarReX, a new method for learning NeRF-based full-body avatars from video data. The learnt avatar not only provides expressive control of the body, hands and the face together, but also supports real-time animation and rendering.
  arXiv  Detail & Related papers  (2023-05-08T15:43:00Z)
• Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization [91.52882218901627]
  We propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing. Our method improves upon photo-realism, geometry, and expression accuracy compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T17:58:40Z)
• HandNeRF: Neural Radiance Fields for Animatable Interacting Hands [122.32855646927013]
  We propose a novel framework to reconstruct accurate appearance and geometry with neural radiance fields (NeRF) for interacting hands. We conduct extensive experiments to verify the merits of our proposed HandNeRF and report a series of state-of-the-art results.
  arXiv  Detail & Related papers  (2023-03-24T06:19:19Z)
• HARP: Personalized Hand Reconstruction from a Monocular RGB Video [37.384221764796095]
  We present HARP, a personalized hand avatar creation approach that takes a short monocular RGB video of a human hand as input. In contrast to the major trend of neural implicit representations, HARP models a hand with a mesh-based parametric hand model. HarP can be directly used in AR/VR applications with real-time rendering capability.
  arXiv  Detail & Related papers  (2022-12-19T15:21:55Z)
• PointAvatar: Deformable Point-based Head Avatars from Videos [103.43941945044294]
  PointAvatar is a deformable point-based representation that disentangles the source color into intrinsic albedo and normal-dependent shading. We show that our method is able to generate animatable 3D avatars using monocular videos from multiple sources.
  arXiv  Detail & Related papers  (2022-12-16T10:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.