CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models

• URL: http://arxiv.org/abs/2412.12093v1
• Date: Mon, 16 Dec 2024 18:58:51 GMT
• Title: CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models
• Authors: Felix Taubner, Ruihang Zhang, Mathieu Tuli, David B. Lindell,
• Abstract summary: CAP4D is an approach that uses a morphable multi-view diffusion model to reconstruct photoreal 4D portrait avatars from any number of reference images.<n>Our approach demonstrates state-of-the-art performance for single-, few-, and multi-image 4D portrait avatar reconstruction.
• Score: 9.622857933809067
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing photorealistic and dynamic portrait avatars from images is essential to many applications including advertising, visual effects, and virtual reality. Depending on the application, avatar reconstruction involves different capture setups and constraints $-$ for example, visual effects studios use camera arrays to capture hundreds of reference images, while content creators may seek to animate a single portrait image downloaded from the internet. As such, there is a large and heterogeneous ecosystem of methods for avatar reconstruction. Techniques based on multi-view stereo or neural rendering achieve the highest quality results, but require hundreds of reference images. Recent generative models produce convincing avatars from a single reference image, but visual fidelity yet lags behind multi-view techniques. Here, we present CAP4D: an approach that uses a morphable multi-view diffusion model to reconstruct photoreal 4D (dynamic 3D) portrait avatars from any number of reference images (i.e., one to 100) and animate and render them in real time. Our approach demonstrates state-of-the-art performance for single-, few-, and multi-image 4D portrait avatar reconstruction, and takes steps to bridge the gap in visual fidelity between single-image and multi-view reconstruction techniques.

Related papers

• FaceCraft4D: Animated 3D Facial Avatar Generation from a Single Image [41.598551483524666]
  We present a novel framework for generating high-quality, animatable 4D avatar from a single image. Our method achieves superior quality compared to the prior art, while maintaining consistency across different viewpoints and expressions.
  arXiv  Detail & Related papers  (2025-04-21T15:40:14Z)
• AvatarArtist: Open-Domain 4D Avatarization [95.63675560402274]
  This work focuses on open-domain 4D avatarization, with the purpose of creating a 4D avatar from a portrait image in an arbitrary style. We select parametric triplanes as the intermediate 4D representation and propose a practical training paradigm that takes advantage of both generative adversarial networks (GANs) and diffusion models.
  arXiv  Detail & Related papers  (2025-03-25T17:59:03Z)
• FRESA: Feedforward Reconstruction of Personalized Skinned Avatars from Few Images [74.86864398919467]
  We present a novel method for reconstructing personalized 3D human avatars with realistic animation from only a few images. We learn a universal prior from over a thousand clothed humans to achieve instant feedforward generation and zero-shot generalization. Our method generates more authentic reconstruction and animation than state-of-the-arts, and can be directly generalized to inputs from casually taken phone photos.
  arXiv  Detail & Related papers  (2025-03-24T23:20:47Z)
• Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior [31.780579293685797]
  We present Vid2Avatar-Pro, a method to create photorealistic and animatable 3D human avatars from monocular in-the-wild videos.
  arXiv  Detail & Related papers  (2025-03-03T14:45:35Z)
• Avat3r: Large Animatable Gaussian Reconstruction Model for High-fidelity 3D Head Avatars [52.439807298140394]
  We present Avat3r, which regresses a high-quality and animatable 3D head avatar from just a few input images. We make Large Reconstruction Models animatable and learn a powerful prior over 3D human heads from a large multi-view video dataset. We increase robustness by feeding input images with different expressions to our model during training, enabling the reconstruction of 3D head avatars from inconsistent inputs.
  arXiv  Detail & Related papers  (2025-02-27T16:00:11Z)
• AniGS: Animatable Gaussian Avatar from a Single Image with Inconsistent Gaussian Reconstruction [26.82525451095629]
  We propose a robust method for 3D reconstruction of inconsistent images, enabling real-time rendering during inference.<n>We recast the reconstruction problem as a 4D task and introduce an efficient 3D modeling approach using 4D Gaussian Splatting.<n>Experiments demonstrate that our method achieves photorealistic, real-time animation of 3D human avatars from in-the-wild images.
  arXiv  Detail & Related papers  (2024-12-03T18:55:39Z)
• GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars [44.8290935585746]
  Photo-realistic and controllable 3D avatars are crucial for various applications such as virtual and mixed reality (VR/MR), telepresence, gaming, and film production. Traditional methods for avatar creation often involve time-consuming scanning and reconstruction processes for each avatar. We propose a text-conditioned generative model that can generate photo-realistic facial avatars of diverse identities.
  arXiv  Detail & Related papers  (2024-08-24T21:25:22Z)
• Instant 3D Human Avatar Generation using Image Diffusion Models [37.45927867788691]
  AvatarPopUp is a method for fast, high quality 3D human avatar generation from different input modalities. Our approach can produce a 3D model in as few as 2 seconds.
  arXiv  Detail & Related papers  (2024-06-11T17:47:27Z)
• Generalizable One-shot Neural Head Avatar [90.50492165284724]
  We present a method that reconstructs and animates a 3D head avatar from a single-view portrait image. We propose a framework that not only generalizes to unseen identities based on a single-view image, but also captures characteristic details within and beyond the face area.
  arXiv  Detail & Related papers  (2023-06-14T22:33:09Z)
• OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Rendering [81.55960827071661]
  Controllability, generalizability and efficiency are the major objectives of constructing face avatars represented by neural implicit field. We propose One-shot Talking face Avatar (OTAvatar), which constructs face avatars by a generalized controllable tri-plane rendering solution.
  arXiv  Detail & Related papers  (2023-03-26T09:12:03Z)
• PointAvatar: Deformable Point-based Head Avatars from Videos [103.43941945044294]
  PointAvatar is a deformable point-based representation that disentangles the source color into intrinsic albedo and normal-dependent shading. We show that our method is able to generate animatable 3D avatars using monocular videos from multiple sources.
  arXiv  Detail & Related papers  (2022-12-16T10:05:31Z)
• MVP-Human Dataset for 3D Human Avatar Reconstruction from Unconstrained Frames [59.37430649840777]
  We present 3D Avatar Reconstruction in the wild (ARwild), which first reconstructs the implicit skinning fields in a multi-level manner. We contribute a large-scale dataset, MVP-Human, which contains 400 subjects, each of which has 15 scans in different poses. Overall, benefits from the specific network architecture and the diverse data, the trained model enables 3D avatar reconstruction from unconstrained frames.
  arXiv  Detail & Related papers  (2022-04-24T03:57:59Z)
• Pixel Codec Avatars [99.36561532588831]
  Pixel Codec Avatars (PiCA) is a deep generative model of 3D human faces. On a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered in realtime in the same scene.
  arXiv  Detail & Related papers  (2021-04-09T23:17:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.