VR Facial Animation for Immersive Telepresence Avatars
- URL: http://arxiv.org/abs/2304.12051v1
- Date: Mon, 24 Apr 2023 12:43:51 GMT
- Title: VR Facial Animation for Immersive Telepresence Avatars
- Authors: Andre Rochow, Max Schwarz, Michael Schreiber, Sven Behnke
- Abstract summary: VR Facial Animation is necessary in applications requiring clear view of the face, even though a VR headset is worn.
We propose a real-time capable pipeline with very fast adaptation for specific operators.
We demonstrate an eye tracking pipeline that can be trained in less than a minute.
- Score: 25.506570225219406
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: VR Facial Animation is necessary in applications requiring clear view of the
face, even though a VR headset is worn. In our case, we aim to animate the face
of an operator who is controlling our robotic avatar system. We propose a
real-time capable pipeline with very fast adaptation for specific operators. In
a quick enrollment step, we capture a sequence of source images from the
operator without the VR headset which contain all the important
operator-specific appearance information. During inference, we then use the
operator keypoint information extracted from a mouth camera and two eye cameras
to estimate the target expression and head pose, to which we map the appearance
of a source still image. In order to enhance the mouth expression accuracy, we
dynamically select an auxiliary expression frame from the captured sequence.
This selection is done by learning to transform the current mouth keypoints
into the source camera space, where the alignment can be determined accurately.
We, furthermore, demonstrate an eye tracking pipeline that can be trained in
less than a minute, a time efficient way to train the whole pipeline given a
dataset that includes only complete faces, show exemplary results generated by
our method, and discuss performance at the ANA Avatar XPRIZE semifinals.
Related papers
- FreeAvatar: Robust 3D Facial Animation Transfer by Learning an Expression Foundation Model [45.0201701977516]
Video-driven 3D facial animation transfer aims to drive avatars to reproduce the expressions of actors.
We propose FreeAvatar, a robust facial animation transfer method that relies solely on our learned expression representation.
arXiv Detail & Related papers (2024-09-20T03:17:01Z) - GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations [54.94362657501809]
We propose a new method to generate highly dynamic and deformable human head avatars from multi-view imagery in real-time.
At the core of our method is a hierarchical representation of head models that allows to capture the complex dynamics of facial expressions and head movements.
We train this coarse-to-fine facial avatar model along with the head pose as a learnable parameter in an end-to-end framework.
arXiv Detail & Related papers (2024-09-18T13:05:43Z) - Universal Facial Encoding of Codec Avatars from VR Headsets [32.60236093340087]
We present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset.
We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency.
arXiv Detail & Related papers (2024-07-17T22:08:15Z) - Real-Time Simulated Avatar from Head-Mounted Sensors [70.41580295721525]
We present SimXR, a method for controlling a simulated avatar from information (headset pose and cameras) obtained from AR / VR headsets.
To synergize headset poses with cameras, we control a humanoid to track headset movement while analyzing input images to decide body movement.
When body parts are seen, the movements of hands and feet will be guided by the images; when unseen, the laws of physics guide the controller to generate plausible motion.
arXiv Detail & Related papers (2024-03-11T16:15:51Z) - Attention-Based VR Facial Animation with Visual Mouth Camera Guidance
for Immersive Telepresence Avatars [19.70403947793871]
We present a hybrid method that uses both keypoints and direct visual guidance from a mouth camera.
Our method generalizes to unseen operators and requires only a quick enrolment step with capture of two short videos.
We highlight how the facial animation contributed to our victory at the ANA Avatar XPRIZE Finals.
arXiv Detail & Related papers (2023-12-15T12:45:11Z) - GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar [48.21353924040671]
We propose to learn person-specific animatable avatars from images without assuming to have access to precise facial expression tracking.
We learn a mapping from 3DMM facial expression parameters to the latent space of the generative model.
With this scheme, we decouple 3D appearance reconstruction and animation control to achieve high fidelity in image synthesis.
arXiv Detail & Related papers (2023-11-22T19:13:00Z) - Towards a Pipeline for Real-Time Visualization of Faces for VR-based
Telepresence and Live Broadcasting Utilizing Neural Rendering [58.720142291102135]
Head-mounted displays (HMDs) for Virtual Reality pose a considerable obstacle for a realistic face-to-face conversation in VR.
We present an approach that focuses on low-cost hardware and can be used on a commodity gaming computer with a single GPU.
arXiv Detail & Related papers (2023-01-04T08:49:51Z) - Unmasking Communication Partners: A Low-Cost AI Solution for Digitally
Removing Head-Mounted Displays in VR-Based Telepresence [62.997667081978825]
Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD)
Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware.
We propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware.
arXiv Detail & Related papers (2020-11-06T23:17:12Z) - Audio- and Gaze-driven Facial Animation of Codec Avatars [149.0094713268313]
We describe the first approach to animate Codec Avatars in real-time using audio and/or eye tracking.
Our goal is to display expressive conversations between individuals that exhibit important social signals.
arXiv Detail & Related papers (2020-08-11T22:28:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.