Towards an objective characterization of an individual's facial movements using Self-Supervised Person-Specific-Models

• URL: http://arxiv.org/abs/2211.08279v1
• Date: Tue, 15 Nov 2022 16:30:24 GMT
• Title: Towards an objective characterization of an individual's facial movements using Self-Supervised Person-Specific-Models
• Authors: Yanis Tazi, Michael Berger, and Winrich A. Freiwald
• Abstract summary: We present a novel training approach to learn facial movements independently of other facial characteristics. One model per individual can learn to extract an embedding of the facial movements independently of the person's identity. We present quantitative and qualitative evidence that this approach is easily scalable and generalizable for new individuals.
• Score: 0.3441021278275805
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Disentangling facial movements from other facial characteristics, particularly from facial identity, remains a challenging task, as facial movements display great variation between individuals. In this paper, we aim to characterize individual-specific facial movements. We present a novel training approach to learn facial movements independently of other facial characteristics, focusing on each individual separately. We propose self-supervised Person-Specific Models (PSMs), in which one model per individual can learn to extract an embedding of the facial movements independently of the person's identity and other structural facial characteristics from unlabeled facial video. These models are trained using encoder-decoder-like architectures. We provide quantitative and qualitative evidence that a PSM learns a meaningful facial embedding that discovers fine-grained movements otherwise not characterized by a General Model (GM), which is trained across individuals and characterizes general patterns of facial movements. We present quantitative and qualitative evidence that this approach is easily scalable and generalizable for new individuals: facial movements knowledge learned on a person can quickly and effectively be transferred to a new person. Lastly, we propose a novel PSM using curriculum temporal learning to leverage the temporal contiguity between video frames. Our code, analysis details, and all pretrained models are available in Github and Supplementary Materials.

Related papers

• MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes [74.82911268630463]
  Talking face generation (TFG) aims to animate a target identity's face to create realistic talking videos. MimicTalk exploits the rich knowledge from a NeRF-based person-agnostic generic model for improving the efficiency and robustness of personalized TFG. Experiments show that our MimicTalk surpasses previous baselines regarding video quality, efficiency, and expressiveness.
  arXiv  Detail & Related papers  (2024-10-09T10:12:37Z)
• Diverse Code Query Learning for Speech-Driven Facial Animation [2.1779479916071067]
  Speech-driven facial animation aims to synthesize lip-synchronized 3D talking faces following the given speech signal. We propose predicting multiple samples conditioned on the same audio signal and then explicitly encouraging sample diversity to address diverse facial animation.
  arXiv  Detail & Related papers  (2024-09-27T21:15:21Z)
• A Generalist FaceX via Learning Unified Facial Representation [77.74407008931486]
  FaceX is a novel facial generalist model capable of handling diverse facial tasks simultaneously. Our versatile FaceX achieves competitive performance compared to elaborate task-specific models on popular facial editing tasks.
  arXiv  Detail & Related papers  (2023-12-31T17:41:48Z)
• VINECS: Video-based Neural Character Skinning [82.39776643541383]
  We propose a fully automated approach for creating a fully rigged character with pose-dependent skinning weights. We show that our approach outperforms state-of-the-art while not relying on dense 4D scans.
  arXiv  Detail & Related papers  (2023-07-03T08:35:53Z)
• CIAO! A Contrastive Adaptation Mechanism for Non-Universal Facial Expression Recognition [80.07590100872548]
  We propose Contrastive Inhibitory Adaptati On (CIAO), a mechanism that adapts the last layer of facial encoders to depict specific affective characteristics on different datasets. CIAO presents an improvement in facial expression recognition performance over six different datasets with very unique affective representations.
  arXiv  Detail & Related papers  (2022-08-10T15:46:05Z)
• Mitigating Bias in Facial Analysis Systems by Incorporating Label Diversity [4.089080285684415]
  We introduce a novel learning method that combines subjective human-based labels and objective annotations based on mathematical definitions of facial traits. Our method successfully mitigates unintended biases, while maintaining significant accuracy on the downstream task.
  arXiv  Detail & Related papers  (2022-04-13T13:17:27Z)
• Learning Facial Representations from the Cycle-consistency of Face [23.23272327438177]
  We introduce cycle-consistency in facial characteristics as free supervisory signal to learn facial representations from unlabeled facial images. The learning is realized by superimposing the facial motion cycle-consistency and identity cycle-consistency constraints. Our approach is competitive with those of existing methods, demonstrating the rich and unique information embedded in the disentangled representations.
  arXiv  Detail & Related papers  (2021-08-07T11:30:35Z)
• I Only Have Eyes for You: The Impact of Masks On Convolutional-Based Facial Expression Recognition [78.07239208222599]
  We evaluate how the recently proposed FaceChannel adapts towards recognizing facial expressions from persons with masks. We also perform specific feature-level visualization to demonstrate how the inherent capabilities of the FaceChannel to learn and combine facial features change when in a constrained social interaction scenario.
  arXiv  Detail & Related papers  (2021-04-16T20:03:30Z)
• Pre-training strategies and datasets for facial representation learning [58.8289362536262]
  We show how to find a universal face representation that can be adapted to several facial analysis tasks and datasets. We systematically investigate two ways of large-scale representation learning applied to faces: supervised and unsupervised pre-training. Our main two findings are: Unsupervised pre-training on completely in-the-wild, uncurated data provides consistent and, in some cases, significant accuracy improvements.
  arXiv  Detail & Related papers  (2021-03-30T17:57:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.