VR Facial Animation for Immersive Telepresence Avatars
- URL: http://arxiv.org/abs/2304.12051v1
- Date: Mon, 24 Apr 2023 12:43:51 GMT
- Title: VR Facial Animation for Immersive Telepresence Avatars
- Authors: Andre Rochow, Max Schwarz, Michael Schreiber, Sven Behnke
- Abstract summary: VR Facial Animation is necessary in applications requiring clear view of the face, even though a VR headset is worn.
We propose a real-time capable pipeline with very fast adaptation for specific operators.
We demonstrate an eye tracking pipeline that can be trained in less than a minute.
- Score: 25.506570225219406
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: VR Facial Animation is necessary in applications requiring clear view of the
face, even though a VR headset is worn. In our case, we aim to animate the face
of an operator who is controlling our robotic avatar system. We propose a
real-time capable pipeline with very fast adaptation for specific operators. In
a quick enrollment step, we capture a sequence of source images from the
operator without the VR headset which contain all the important
operator-specific appearance information. During inference, we then use the
operator keypoint information extracted from a mouth camera and two eye cameras
to estimate the target expression and head pose, to which we map the appearance
of a source still image. In order to enhance the mouth expression accuracy, we
dynamically select an auxiliary expression frame from the captured sequence.
This selection is done by learning to transform the current mouth keypoints
into the source camera space, where the alignment can be determined accurately.
We, furthermore, demonstrate an eye tracking pipeline that can be trained in
less than a minute, a time efficient way to train the whole pipeline given a
dataset that includes only complete faces, show exemplary results generated by
our method, and discuss performance at the ANA Avatar XPRIZE semifinals.
Related papers
- Universal Facial Encoding of Codec Avatars from VR Headsets [32.60236093340087]
We present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset.
We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency.
arXiv Detail & Related papers (2024-07-17T22:08:15Z) - Real-Time Simulated Avatar from Head-Mounted Sensors [70.41580295721525]
We present SimXR, a method for controlling a simulated avatar from information (headset pose and cameras) obtained from AR / VR headsets.
To synergize headset poses with cameras, we control a humanoid to track headset movement while analyzing input images to decide body movement.
When body parts are seen, the movements of hands and feet will be guided by the images; when unseen, the laws of physics guide the controller to generate plausible motion.
arXiv Detail & Related papers (2024-03-11T16:15:51Z) - Attention-Based VR Facial Animation with Visual Mouth Camera Guidance
for Immersive Telepresence Avatars [19.70403947793871]
We present a hybrid method that uses both keypoints and direct visual guidance from a mouth camera.
Our method generalizes to unseen operators and requires only a quick enrolment step with capture of two short videos.
We highlight how the facial animation contributed to our victory at the ANA Avatar XPRIZE Finals.
arXiv Detail & Related papers (2023-12-15T12:45:11Z) - GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar [48.21353924040671]
We propose to learn person-specific animatable avatars from images without assuming to have access to precise facial expression tracking.
We learn a mapping from 3DMM facial expression parameters to the latent space of the generative model.
With this scheme, we decouple 3D appearance reconstruction and animation control to achieve high fidelity in image synthesis.
arXiv Detail & Related papers (2023-11-22T19:13:00Z) - AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars [84.85009267371218]
We propose AvatarStudio, a text-based method for editing the appearance of a dynamic full head avatar.
Our approach builds on existing work to capture dynamic performances of human heads using neural field (NeRF) and edits this representation with a text-to-image diffusion model.
Our method edits the full head in a canonical space, and then propagates these edits to remaining time steps via a pretrained deformation network.
arXiv Detail & Related papers (2023-06-01T11:06:01Z) - 4D Agnostic Real-Time Facial Animation Pipeline for Desktop Scenarios [8.274472944075713]
We present a high-precision real-time facial animation pipeline suitable for animators to use on their desktops.
System enables animators to create high-quality facial animations with ease and speed.
Our approach has the potential to revolutionize the way facial animation is done in the entertainment industry.
arXiv Detail & Related papers (2023-04-06T01:32:58Z) - Towards a Pipeline for Real-Time Visualization of Faces for VR-based
Telepresence and Live Broadcasting Utilizing Neural Rendering [58.720142291102135]
Head-mounted displays (HMDs) for Virtual Reality pose a considerable obstacle for a realistic face-to-face conversation in VR.
We present an approach that focuses on low-cost hardware and can be used on a commodity gaming computer with a single GPU.
arXiv Detail & Related papers (2023-01-04T08:49:51Z) - Unmasking Communication Partners: A Low-Cost AI Solution for Digitally
Removing Head-Mounted Displays in VR-Based Telepresence [62.997667081978825]
Face-to-face conversation in Virtual Reality (VR) is a challenge when participants wear head-mounted displays (HMD)
Past research has shown that high-fidelity face reconstruction with personal avatars in VR is possible under laboratory conditions with high-cost hardware.
We propose one of the first low-cost systems for this task which uses only open source, free software and affordable hardware.
arXiv Detail & Related papers (2020-11-06T23:17:12Z) - Audio- and Gaze-driven Facial Animation of Codec Avatars [149.0094713268313]
We describe the first approach to animate Codec Avatars in real-time using audio and/or eye tracking.
Our goal is to display expressive conversations between individuals that exhibit important social signals.
arXiv Detail & Related papers (2020-08-11T22:28:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.