Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior

• URL: http://arxiv.org/abs/2503.01610v1
• Date: Mon, 03 Mar 2025 14:45:35 GMT
• Title: Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior
• Authors: Chen Guo, Junxuan Li, Yash Kant, Yaser Sheikh, Shunsuke Saito, Chen Cao,
• Abstract summary: We present Vid2Avatar-Pro, a method to create photorealistic and animatable 3D human avatars from monocular in-the-wild videos.
• Score: 31.780579293685797
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Vid2Avatar-Pro, a method to create photorealistic and animatable 3D human avatars from monocular in-the-wild videos. Building a high-quality avatar that supports animation with diverse poses from a monocular video is challenging because the observation of pose diversity and view points is inherently limited. The lack of pose variations typically leads to poor generalization to novel poses, and avatars can easily overfit to limited input view points, producing artifacts and distortions from other views. In this work, we address these limitations by leveraging a universal prior model (UPM) learned from a large corpus of multi-view clothed human performance capture data. We build our representation on top of expressive 3D Gaussians with canonical front and back maps shared across identities. Once the UPM is learned to accurately reproduce the large-scale multi-view human images, we fine-tune the model with an in-the-wild video via inverse rendering to obtain a personalized photorealistic human avatar that can be faithfully animated to novel human motions and rendered from novel views. The experiments show that our approach based on the learned universal prior sets a new state-of-the-art in monocular avatar reconstruction by substantially outperforming existing approaches relying only on heuristic regularization or a shape prior of minimally clothed bodies (e.g., SMPL) on publicly available datasets.

Related papers

• FRESA: Feedforward Reconstruction of Personalized Skinned Avatars from Few Images [74.86864398919467]
  We present a novel method for reconstructing personalized 3D human avatars with realistic animation from only a few images. We learn a universal prior from over a thousand clothed humans to achieve instant feedforward generation and zero-shot generalization. Our method generates more authentic reconstruction and animation than state-of-the-arts, and can be directly generalized to inputs from casually taken phone photos.
  arXiv  Detail & Related papers  (2025-03-24T23:20:47Z)
• WonderHuman: Hallucinating Unseen Parts in Dynamic 3D Human Reconstruction [51.22641018932625]
  We present WonderHuman to reconstruct dynamic human avatars from a monocular video for high-fidelity novel view synthesis.<n>Our method achieves SOTA performance in producing photorealistic renderings from the given monocular video.
  arXiv  Detail & Related papers  (2025-02-03T04:43:41Z)
• Deblur-Avatar: Animatable Avatars from Motion-Blurred Monocular Videos [64.10307207290039]
  De-Avatar is a novel framework for modeling high-fidelity, animatable 3D human avatars from motion-blurred monocular video inputs.
  arXiv  Detail & Related papers  (2025-01-23T02:31:57Z)
• IDOL: Instant Photorealistic 3D Human Creation from a Single Image [47.51400420928373]
  This work rethinks the task from the perspectives of dataset, model, and representation.<n>We introduce a large-scale HUman-centric GEnerated dataset, HuGe100K, consisting of 100K diverse, photorealistic sets of human images.<n>We develop a scalable feed-forward transformer model to predict a 3D human Gaussian representation in a uniform space from a given human image.
  arXiv  Detail & Related papers  (2024-12-19T15:43:05Z)
• Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
  We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence. ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images. Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
  arXiv  Detail & Related papers  (2023-12-22T20:56:46Z)
• Relightable Neural Actor with Intrinsic Decomposition and Pose Control [80.06094206522668]
  We propose Relightable Neural Actor, a new video-based method for learning a pose-driven neural human model that can be relighted. For training, our method solely requires a multi-view recording of the human under a known, but static lighting condition. To evaluate our approach in real-world scenarios, we collect a new dataset with four identities recorded under different light conditions, indoors and outdoors.
  arXiv  Detail & Related papers  (2023-12-18T14:30:13Z)
• Reality's Canvas, Language's Brush: Crafting 3D Avatars from Monocular Video [14.140380599168628]
  ReCaLaB is a pipeline that learns high-fidelity 3D human avatars from just a single RGB video. A pose-conditioned NeRF is optimized to volumetrically represent a human subject in canonical T-pose. An image-conditioned diffusion model thereby helps to animate appearance and pose of the 3D avatar to create video sequences with previously unseen human motion.
  arXiv  Detail & Related papers  (2023-12-08T01:53:06Z)
• Towards 4D Human Video Stylization [56.33756124829298]
  We present a first step towards 4D (3D and time) human video stylization, which addresses style transfer, novel view synthesis and human animation. We leverage Neural Radiance Fields (NeRFs) to represent videos, conducting stylization in the rendered feature space. Our framework uniquely extends its capabilities to accommodate novel poses and viewpoints, making it a versatile tool for creative human video stylization.
  arXiv  Detail & Related papers  (2023-12-07T08:58:33Z)
• Animatable and Relightable Gaussians for High-fidelity Human Avatar Modeling [47.1427140235414]
  We introduce a new avatar representation that leverages powerful 2D CNNs and 3D Gaussian splatting to create high-fidelity avatars. Our method can create lifelike avatars with dynamic, realistic, generalized and relightable appearances.
  arXiv  Detail & Related papers  (2023-11-27T18:59:04Z)
• PointAvatar: Deformable Point-based Head Avatars from Videos [103.43941945044294]
  PointAvatar is a deformable point-based representation that disentangles the source color into intrinsic albedo and normal-dependent shading. We show that our method is able to generate animatable 3D avatars using monocular videos from multiple sources.
  arXiv  Detail & Related papers  (2022-12-16T10:05:31Z)
• AvatarGen: a 3D Generative Model for Animatable Human Avatars [108.11137221845352]
  AvatarGen is the first method that enables not only non-rigid human generation with diverse appearance but also full control over poses and viewpoints. To model non-rigid dynamics, it introduces a deformation network to learn pose-dependent deformations in the canonical space. Our method can generate animatable human avatars with high-quality appearance and geometry modeling, significantly outperforming previous 3D GANs.
  arXiv  Detail & Related papers  (2022-08-01T01:27:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.