FacialMotionID: Identifying Users of Mixed Reality Headsets using Abstract Facial Motion Representations

• URL: http://arxiv.org/abs/2507.11138v1
• Date: Tue, 15 Jul 2025 09:40:49 GMT
• Title: FacialMotionID: Identifying Users of Mixed Reality Headsets using Abstract Facial Motion Representations
• Authors: Adriano Castro, Simon Hanisch, Matin Fallahi, Thorsten Strufe,
• Abstract summary: Facial motion capture in mixed reality headsets enables real-time avatar animation, allowing users to convey non-verbal cues during virtual interactions.<n>As facial motion data constitutes a behavioral biometric, its use raises novel privacy concerns.<n>We conducted a study with 116 participants using three types of headsets across three sessions, collecting facial, eye, and head motion data during verbal and non-verbal tasks.<n>Our analysis shows that individuals can be re-identified from this data with up to 98% balanced accuracy, are even identifiable across device types, and that emotional states can be inferred with up to 86% accuracy.
• Score: 2.9136421025415213
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Facial motion capture in mixed reality headsets enables real-time avatar animation, allowing users to convey non-verbal cues during virtual interactions. However, as facial motion data constitutes a behavioral biometric, its use raises novel privacy concerns. With mixed reality systems becoming more immersive and widespread, understanding whether face motion data can lead to user identification or inference of sensitive attributes is increasingly important. To address this, we conducted a study with 116 participants using three types of headsets across three sessions, collecting facial, eye, and head motion data during verbal and non-verbal tasks. The data used is not raw video, but rather, abstract representations that are used to animate digital avatars. Our analysis shows that individuals can be re-identified from this data with up to 98% balanced accuracy, are even identifiable across device types, and that emotional states can be inferred with up to 86% accuracy. These results underscore the potential privacy risks inherent in face motion tracking in mixed reality environments.

Related papers

• Is It Really You? Exploring Biometric Verification Scenarios in Photorealistic Talking-Head Avatar Videos [12.12643642515884]
  An attacker can steal a user's avatar, preserving his appearance and voice, making it nearly impossible to detect its usage by sight or sound alone.<n>Our main question is whether an individual's facial motion patterns can serve as reliable behavioral biometrics to verify their identity when the avatar's visual appearance is a facsimile of its owner.<n> Experimental results demonstrate that facial motion landmarks enable meaningful identity verification with AUC values approaching 80%.
  arXiv  Detail & Related papers  (2025-08-01T16:23:27Z)
• VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis [70.76837748695841]
  We propose VisualSpeaker, a novel method that bridges the gap using photorealistic differentiable rendering, supervised by visual speech recognition, for improved 3D facial animation.<n>Our contribution is a perceptual lip-reading loss, derived by passing 3D Gaussian Splatting avatar renders through a pre-trained Visual Automatic Speech Recognition model during training.<n> Evaluation on the MEAD dataset demonstrates that VisualSpeaker improves both the standard Lip Vertex Error metric by 56.1% and the perceptual quality of the generated animations, while retaining the controllability of mesh-driven animation.
  arXiv  Detail & Related papers  (2025-07-08T15:04:17Z)
• Universal Facial Encoding of Codec Avatars from VR Headsets [32.60236093340087]
  We present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency.
  arXiv  Detail & Related papers  (2024-07-17T22:08:15Z)
• AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding [24.486705010561067]
  The paper introduces AniTalker, a framework designed to generate lifelike talking faces from a single portrait. AniTalker effectively captures a wide range of facial dynamics, including subtle expressions and head movements.
  arXiv  Detail & Related papers  (2024-05-06T02:32:41Z)
• From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations [107.88375243135579]
  Given speech audio, we output multiple possibilities of gestural motion for an individual, including face, body, and hands. We visualize the generated motion using highly photorealistic avatars that can express crucial nuances in gestures. Experiments show our model generates appropriate and diverse gestures, outperforming both diffusion- and VQ-only methods.
  arXiv  Detail & Related papers  (2024-01-03T18:55:16Z)
• Attention-Based VR Facial Animation with Visual Mouth Camera Guidance for Immersive Telepresence Avatars [19.70403947793871]
  We present a hybrid method that uses both keypoints and direct visual guidance from a mouth camera. Our method generalizes to unseen operators and requires only a quick enrolment step with capture of two short videos. We highlight how the facial animation contributed to our victory at the ANA Avatar XPRIZE Finals.
  arXiv  Detail & Related papers  (2023-12-15T12:45:11Z)
• Physics-based Motion Retargeting from Sparse Inputs [73.94570049637717]
  Commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available.
  arXiv  Detail & Related papers  (2023-07-04T21:57:05Z)
• Unique Identification of 50,000+ Virtual Reality Users from Head & Hand Motion Data [58.27542320038834]
  We show that a large number of real VR users can be uniquely and reliably identified across multiple sessions using just their head and hand motion. After training a classification model on 5 minutes of data per person, a user can be uniquely identified amongst the entire pool of 50,000+ with 94.33% accuracy from 100 seconds of motion. This work is the first to truly demonstrate the extent to which biomechanics may serve as a unique identifier in VR, on par with widely used biometrics such as facial or fingerprint recognition.
  arXiv  Detail & Related papers  (2023-02-17T15:05:18Z)
• Towards Inclusive HRI: Using Sim2Real to Address Underrepresentation in Emotion Expression Recognition [5.819149317261972]
  We aim to build a system that can perceive humans in a more transparent and inclusive manner. We use a Sim2Real approach in which we use a suite of 3D simulated human models. By augmenting a small dynamic emotional expression dataset with a synthetic dataset containing 4536 samples, we achieved an improvement in accuracy of 15%.
  arXiv  Detail & Related papers  (2022-08-15T23:37:13Z)
• Audio- and Gaze-driven Facial Animation of Codec Avatars [149.0094713268313]
  We describe the first approach to animate Codec Avatars in real-time using audio and/or eye tracking. Our goal is to display expressive conversations between individuals that exhibit important social signals.
  arXiv  Detail & Related papers  (2020-08-11T22:28:48Z)
• Investigating the Impact of Inclusion in Face Recognition Training Data on Individual Face Identification [93.5538147928669]
  We audit ArcFace, a state-of-the-art, open source face recognition system, in a large-scale face identification experiment with more than one million distractor images. We find a Rank-1 face identification accuracy of 79.71% for individuals present in the model's training data and an accuracy of 75.73% for those not present.
  arXiv  Detail & Related papers  (2020-01-09T15:50:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.