Facial Prior Based First Order Motion Model for Micro-expression Generation

• URL: http://arxiv.org/abs/2308.04536v1
• Date: Tue, 8 Aug 2023 18:57:03 GMT
• Title: Facial Prior Based First Order Motion Model for Micro-expression Generation
• Authors: Yi Zhang, Youjun Zhao, Yuhang Wen, Zixuan Tang, Xinhua Xu, Mengyuan Liu
• Abstract summary: This paper tries to formulate a new task called micro-expression generation. It combines the first order motion model with facial prior knowledge. Given a target face, we intend to drive the face to generate micro-expression videos according to the motion patterns of source videos.
• Score: 11.27890186026442
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spotting facial micro-expression from videos finds various potential applications in fields including clinical diagnosis and interrogation, meanwhile this task is still difficult due to the limited scale of training data. To solve this problem, this paper tries to formulate a new task called micro-expression generation and then presents a strong baseline which combines the first order motion model with facial prior knowledge. Given a target face, we intend to drive the face to generate micro-expression videos according to the motion patterns of source videos. Specifically, our new model involves three modules. First, we extract facial prior features from a region focusing module. Second, we estimate facial motion using key points and local affine transformations with a motion prediction module. Third, expression generation module is used to drive the target face to generate videos. We train our model on public CASME II, SAMM and SMIC datasets and then use the model to generate new micro-expression videos for evaluation. Our model achieves the first place in the Facial Micro-Expression Challenge 2021 (MEGC2021), where our superior performance is verified by three experts with Facial Action Coding System certification. Source code is provided in https://github.com/Necolizer/Facial-Prior-Based-FOMM.

Related papers

• Face-MLLM: A Large Face Perception Model [53.9441375205716]
  multimodal large language models (MLLMs) have achieved promising results on a wide range of vision-language tasks, but their ability to perceive and understand human faces is rarely explored. In this work, we comprehensively evaluate existing MLLMs on face perception tasks. Our model surpasses previous MLLMs on five famous face perception tasks.
  arXiv  Detail & Related papers  (2024-10-28T04:19:32Z)
• MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes [74.82911268630463]
  Talking face generation (TFG) aims to animate a target identity's face to create realistic talking videos. MimicTalk exploits the rich knowledge from a NeRF-based person-agnostic generic model for improving the efficiency and robustness of personalized TFG. Experiments show that our MimicTalk surpasses previous baselines regarding video quality, efficiency, and expressiveness.
  arXiv  Detail & Related papers  (2024-10-09T10:12:37Z)
• GSmoothFace: Generalized Smooth Talking Face Generation via Fine Grained 3D Face Guidance [83.43852715997596]
  GSmoothFace is a novel two-stage generalized talking face generation model guided by a fine-grained 3d face model. It can synthesize smooth lip dynamics while preserving the speaker's identity. Both quantitative and qualitative experiments confirm the superiority of our method in terms of realism, lip synchronization, and visual quality.
  arXiv  Detail & Related papers  (2023-12-12T16:00:55Z)
• 4D Facial Expression Diffusion Model [3.507793603897647]
  We introduce a generative framework for generating 3D facial expression sequences. It is composed of two tasks: Learning the generative model that is trained over a set of 3D landmark sequences, and Generating 3D mesh sequences of an input facial mesh driven by the generated landmark sequences. Experiments show that our model has learned to generate realistic, quality expressions solely from the dataset of relatively small size, improving over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-29T11:50:21Z)
• MAGVIT: Masked Generative Video Transformer [129.50814875955444]
  We introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model. A single MAGVIT model supports ten diverse generation tasks and generalizes across videos from different visual domains.
  arXiv  Detail & Related papers  (2022-12-10T04:26:32Z)
• Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN [1.279257604152629]
  We present a novel approach for generating videos of the six basic facial expressions. Our approach is based on Spatio-temporal Conal GANs, that are known to model both content and motion in the same network. The code and the pre-trained model will soon be made publicly available.
  arXiv  Detail & Related papers  (2022-10-20T11:54:32Z)
• Masked Motion Encoding for Self-Supervised Video Representation Learning [84.24773072241945]
  We present Masked Motion MME, a new pre-training paradigm that reconstructs both appearance and motion information to explore temporal clues. Motivated by the fact that human is able to recognize an action by tracking objects' position changes and shape changes, we propose to reconstruct a motion trajectory that represents these two kinds of change in the masked regions. Pre-trained with our MME paradigm, the model is able to anticipate long-term and fine-grained motion details.
  arXiv  Detail & Related papers  (2022-10-12T11:19:55Z)
• MMNet: Muscle motion-guided network for micro-expression recognition [2.032432845751978]
  We propose a robust micro-expression recognition framework, namely muscle motion-guided network (MMNet) Specifically, a continuous attention (CA) block is introduced to focus on modeling local subtle muscle motion patterns with little identity information. Our approach outperforms state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2022-01-14T04:05:49Z)
• A Dynamic 3D Spontaneous Micro-expression Database: Establishment and Evaluation [14.994232615123337]
  Micro-expressions are spontaneous, unconscious facial movements that show people's true inner emotions. The occurrence of an expression can arouse spatial deformation of the face. We propose a new micro-expression database containing 2D video sequences and 3D point clouds sequences.
  arXiv  Detail & Related papers  (2021-07-31T07:04:16Z)
• Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation [75.59415852802958]
  Shape-My-Face (SMF) is a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model. Our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets.
  arXiv  Detail & Related papers  (2020-12-16T20:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.