PrismAvatar: Real-time animated 3D neural head avatars on edge devices

• URL: http://arxiv.org/abs/2502.07030v1
• Date: Mon, 10 Feb 2025 20:50:12 GMT
• Title: PrismAvatar: Real-time animated 3D neural head avatars on edge devices
• Authors: Prashant Raina, Felix Taubner, Mathieu Tuli, Eu Wern Teh, Kevin Ferreira,
• Abstract summary: We present PrismAvatar, a 3D head avatar model which is designed specifically to enable real-time animation and rendering on resource-constrained edge devices. In addition to running at 60 fps with low memory usage on mobile devices, we find that our trained models have comparable quality to state-of-the-art 3D avatar models on desktop devices.
• Score: 2.446672595462589
• License:
• Abstract: We present PrismAvatar: a 3D head avatar model which is designed specifically to enable real-time animation and rendering on resource-constrained edge devices, while still enjoying the benefits of neural volumetric rendering at training time. By integrating a rigged prism lattice with a 3D morphable head model, we use a hybrid rendering model to simultaneously reconstruct a mesh-based head and a deformable NeRF model for regions not represented by the 3DMM. We then distill the deformable NeRF into a rigged mesh and neural textures, which can be animated and rendered efficiently within the constraints of the traditional triangle rendering pipeline. In addition to running at 60 fps with low memory usage on mobile devices, we find that our trained models have comparable quality to state-of-the-art 3D avatar models on desktop devices.

Related papers

• Generating Editable Head Avatars with 3D Gaussian GANs [57.51487984425395]
  Traditional 3D-aware generative adversarial networks (GANs) achieve photorealistic and view-consistent 3D head synthesis. We propose a novel approach that enhances the editability and animation control of 3D head avatars by incorporating 3D Gaussian Splatting (3DGS) as an explicit 3D representation. Our approach delivers high-quality 3D-aware synthesis with state-of-the-art controllability.
  arXiv  Detail & Related papers  (2024-12-26T10:10:03Z)
• SqueezeMe: Mobile-Ready Distillation of Gaussian Full-Body Avatars [19.249226899376943]
  We present SqueezeMe, a framework to convert high-fidelity 3D Gaussian full-body avatars into a lightweight representation. We achieve, for the first time, simultaneous animation and rendering of 3 Gaussian avatars in real-time (72 FPS) on a Meta Quest 3 VR headset.
  arXiv  Detail & Related papers  (2024-12-19T18:46:55Z)
• NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fields [57.617972778377215]
  We show how to generate effective 3D representations from posed RGB images. We pretrain this representation at scale on our proposed curated posed-RGB data, totaling over 1.8 million images. Our novel self-supervised pretraining for NeRFs, NeRF-MAE, scales remarkably well and improves performance on various challenging 3D tasks.
  arXiv  Detail & Related papers  (2024-04-01T17:59:55Z)
• PSAvatar: A Point-based Shape Model for Real-Time Head Avatar Animation with 3D Gaussian Splatting [17.78639236586134]
  PSAvatar is a novel framework for animatable head avatar creation. It employs 3D Gaussian for fine detail representation and high fidelity rendering. We show that PSAvatar can reconstruct high-fidelity head avatars of a variety of subjects and the avatars can be animated in real-time.
  arXiv  Detail & Related papers  (2024-01-23T16:40:47Z)
• FLARE: Fast Learning of Animatable and Relightable Mesh Avatars [64.48254296523977]
  Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. We introduce FLARE, a technique that enables the creation of animatable and relightable avatars from a single monocular video.
  arXiv  Detail & Related papers  (2023-10-26T16:13:00Z)
• AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels [33.085274792188756]
  We propose AvatarMAV, a fast 3D head avatar reconstruction method using Motion-Aware Neural Voxels. AvatarMAV is the first to model both the canonical appearance and the decoupled expression motion by neural voxels for head avatar. The proposed AvatarMAV can recover photo-realistic head avatars in just 5 minutes, which is significantly faster than the state-of-the-art facial reenactment methods.
  arXiv  Detail & Related papers  (2022-11-23T18:49:31Z)
• DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars [92.37436369781692]
  We present DRaCoN, a framework for learning full-body volumetric avatars. It exploits the advantages of both the 2D and 3D neural rendering techniques. Experiments on the challenging ZJU-MoCap and Human3.6M datasets indicate that DRaCoN outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T17:59:15Z)
• HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars [65.82222842213577]
  We propose a novel neural rendering pipeline, which synthesizes virtual human avatars from arbitrary poses efficiently and at high quality. First, we learn to encode articulated human motions on a dense UV manifold of the human body surface. We then leverage the encoded information on the UV manifold to construct a 3D volumetric representation.
  arXiv  Detail & Related papers  (2021-12-19T17:34:15Z)
• HeadNeRF: A Real-time NeRF-based Parametric Head Model [39.240265611700735]
  HeadNeRF is a novel NeRF-based parametric head model that integrates the neural radiance field to the parametric representation of the human head. It can render high fidelity head images in real-time, and supports directly controlling the generated images' rendering pose and various semantic attributes.
  arXiv  Detail & Related papers  (2021-12-10T16:10:13Z)
• SMPLpix: Neural Avatars from 3D Human Models [56.85115800735619]
  We bridge the gap between classic rendering and the latest generative networks operating in pixel space. We train a network that directly converts a sparse set of 3D mesh vertices into photorealistic images. We show the advantage over conventional differentiables both in terms of the level of photorealism and rendering efficiency.
  arXiv  Detail & Related papers  (2020-08-16T10:22:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.