MegaPortraits: One-shot Megapixel Neural Head Avatars

• URL: http://arxiv.org/abs/2207.07621v2
• Date: Tue, 28 Mar 2023 10:58:12 GMT
• Title: MegaPortraits: One-shot Megapixel Neural Head Avatars
• Authors: Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky and Egor Zakharov
• Abstract summary: We propose a set of new neural architectures and training methods that can leverage both medium-resolution video data and high-resolution image data. We show how a trained high-resolution neural avatar model can be distilled into a lightweight student model which runs in real-time. Real-time operation and identity lock are essential for many practical applications head avatar systems.
• Score: 7.05068904295608
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this work, we advance the neural head avatar technology to the megapixel resolution while focusing on the particularly challenging task of cross-driving synthesis, i.e., when the appearance of the driving image is substantially different from the animated source image. We propose a set of new neural architectures and training methods that can leverage both medium-resolution video data and high-resolution image data to achieve the desired levels of rendered image quality and generalization to novel views and motion. We demonstrate that suggested architectures and methods produce convincing high-resolution neural avatars, outperforming the competitors in the cross-driving scenario. Lastly, we show how a trained high-resolution neural avatar model can be distilled into a lightweight student model which runs in real-time and locks the identities of neural avatars to several dozens of pre-defined source images. Real-time operation and identity lock are essential for many practical applications head avatar systems.

Related papers

• GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations [54.94362657501809]
  We propose a new method to generate highly dynamic and deformable human head avatars from multi-view imagery in real-time. At the core of our method is a hierarchical representation of head models that allows to capture the complex dynamics of facial expressions and head movements. We train this coarse-to-fine facial avatar model along with the head pose as a learnable parameter in an end-to-end framework.
  arXiv  Detail & Related papers  (2024-09-18T13:05:43Z)
• BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis [7.485318043174123]
  We introduce BakedAvatar, a novel representation for real-time neural head avatar. Our approach extracts layered meshes from learned isosurfaces of the head and computes expression-, pose-, and view-dependent appearances. Experimental results demonstrate that our representation generates photorealistic results of comparable quality to other state-the-art methods.
  arXiv  Detail & Related papers  (2023-11-09T17:05:53Z)
• Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images [65.41966114373373]
  We present an improved solution to the neural image-based rendering problem in computer vision. The proposed approach could synthesize a realistic image of the scene from a novel viewpoint at test time.
  arXiv  Detail & Related papers  (2023-11-08T08:18:23Z)
• Neural Point-based Volumetric Avatar: Surface-guided Neural Points for Efficient and Photorealistic Volumetric Head Avatar [62.87222308616711]
  We propose fullname (name), a method that adopts the neural point representation and the neural volume rendering process. Specifically, the neural points are strategically constrained around the surface of the target expression via a high-resolution UV displacement map. By design, our name is better equipped to handle topologically changing regions and thin structures while also ensuring accurate expression control when animating avatars.
  arXiv  Detail & Related papers  (2023-07-11T03:40:10Z)
• NOFA: NeRF-based One-shot Facial Avatar Reconstruction [45.11455702291703]
  3D facial avatar reconstruction has been a significant research topic in computer graphics and computer vision. We propose a one-shot 3D facial avatar reconstruction framework that only requires a single source image to reconstruct a high-fidelity 3D facial avatar.
  arXiv  Detail & Related papers  (2023-07-07T07:58:18Z)
• Instant Volumetric Head Avatars [20.782425305421505]
  We present Instant Volumetric Head Avatars (INSTA), a novel approach for reconstructing photo-realistic digital avatars instantaneously. Our pipeline is trained on a single monocular RGB portrait video that observes the subject under different expressions and views. INSTA reconstructs a digital avatar in less than 10 minutes on modern GPU hardware, which is orders of magnitude faster than previous solutions.
  arXiv  Detail & Related papers  (2022-11-22T18:59:46Z)
• Fast Training of Neural Lumigraph Representations using Meta Learning [109.92233234681319]
  We develop a new neural rendering approach with the goal of quickly learning a high-quality representation which can also be rendered in real-time. Our approach, MetaNLR++, accomplishes this by using a unique combination of a neural shape representation and 2D CNN-based image feature extraction, aggregation, and re-projection. We show that MetaNLR++ achieves similar or better photorealistic novel view synthesis results in a fraction of the time that competing methods require.
  arXiv  Detail & Related papers  (2021-06-28T18:55:50Z)
• MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images [60.56518548286836]
  To generate realistic cloth deformations from novel input poses, watertight meshes or dense full-body scans are usually needed as inputs. We propose an approach that can quickly generate realistic clothed human avatars, represented as controllable neural SDFs, given only monocular depth images.
  arXiv  Detail & Related papers  (2021-06-22T17:30:12Z)
• Pixel Codec Avatars [99.36561532588831]
  Pixel Codec Avatars (PiCA) is a deep generative model of 3D human faces. On a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered in realtime in the same scene.
  arXiv  Detail & Related papers  (2021-04-09T23:17:36Z)
• Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars [16.92378994798985]
  We propose a neural rendering-based system that creates head avatars from a single photograph. We compare our system to analogous state-of-the-art systems in terms of visual quality and speed.
  arXiv  Detail & Related papers  (2020-08-24T03:23:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.