A Pre-trained Audio-Visual Transformer for Emotion Recognition

• URL: http://arxiv.org/abs/2201.09165v1
• Date: Sun, 23 Jan 2022 03:09:16 GMT
• Title: A Pre-trained Audio-Visual Transformer for Emotion Recognition
• Abstract summary: We introduce a pretrained audio-visual Transformer trained on more than 500k utterances from nearly 4000 celebrities. The model aims to capture and extract useful information from the interactions between human facial and auditory behaviors.
• Score: 5.321281630718918
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we introduce a pretrained audio-visual Transformer trained on more than 500k utterances from nearly 4000 celebrities from the VoxCeleb2 dataset for human behavior understanding. The model aims to capture and extract useful information from the interactions between human facial and auditory behaviors, with application in emotion recognition. We evaluate the model performance on two datasets, namely CREMAD-D (emotion classification) and MSP-IMPROV (continuous emotion regression). Experimental results show that fine-tuning the pre-trained model helps improving emotion classification accuracy by 5-7% and Concordance Correlation Coefficients (CCC) in continuous emotion recognition by 0.03-0.09 compared to the same model trained from scratch. We also demonstrate the robustness of finetuning the pre-trained model in a low-resource setting. With only 10% of the original training set provided, fine-tuning the pre-trained model can lead to at least 10% better emotion recognition accuracy and a CCC score improvement by at least 0.1 for continuous emotion recognition.

Related papers

• Personalization of Affective Models to Enable Neuropsychiatric Digital Precision Health Interventions: A Feasibility Study [0.820591242098274]
  Mobile digital therapeutics for autism spectrum disorder (ASD) often target emotion recognition and evocation. Here, we explore the potential of model personalization, or training a single emotion recognition model per person. We conducted experiments on the Emognition dataset, a video dataset of human subjects evoking a series of emotions.
  arXiv  Detail & Related papers  (2023-09-21T07:16:50Z)
• Self-Supervised Learning for Audio-Based Emotion Recognition [1.7598252755538808]
  Self-supervised learning is a family of methods which can learn despite a scarcity of supervised labels. We have applied self-supervised learning pre-training to the classification of emotions from the CMU- MOSEI's acoustic modality. We find that self-supervised learning consistently improves the performance of the model across all metrics.
  arXiv  Detail & Related papers  (2023-07-23T14:40:50Z)
• Computer Vision Estimation of Emotion Reaction Intensity in the Wild [1.5481864635049696]
  We describe our submission to the newly introduced Emotional Reaction Intensity (ERI) Estimation challenge. We developed four deep neural networks trained in the visual domain and a multimodal model trained with both visual and audio features to predict emotion reaction intensity.
  arXiv  Detail & Related papers  (2023-03-19T19:09:41Z)
• Revisiting Classifier: Transferring Vision-Language Models for Video Recognition [102.93524173258487]
  Transferring knowledge from task-agnostic pre-trained deep models for downstream tasks is an important topic in computer vision research. In this study, we focus on transferring knowledge for video classification tasks. We utilize the well-pretrained language model to generate good semantic target for efficient transferring learning.
  arXiv  Detail & Related papers  (2022-07-04T10:00:47Z)
• Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation [6.382013662443799]
  Estimating dimensional emotions from acoustic speech signals is challenging. We show that pre-trained acoustic models can improve valence estimation from speech. We report new state-of-the-art "text-free" acoustic-only dimensional emotion estimation.
  arXiv  Detail & Related papers  (2022-07-02T17:34:44Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition [118.73025093045652]
  We propose a pre-training model textbfMEmoBERT for multimodal emotion recognition. Unlike the conventional "pre-train, finetune" paradigm, we propose a prompt-based method that reformulates the downstream emotion classification task as a masked text prediction. Our proposed MEmoBERT significantly enhances emotion recognition performance.
  arXiv  Detail & Related papers  (2021-10-27T09:57:00Z)
• STAR: Sparse Transformer-based Action Recognition [61.490243467748314]
  This work proposes a novel skeleton-based human action recognition model with sparse attention on the spatial dimension and segmented linear attention on the temporal dimension of data. Experiments show that our model can achieve comparable performance while utilizing much less trainable parameters and achieve high speed in training and inference.
  arXiv  Detail & Related papers  (2021-07-15T02:53:11Z)
• Recognizing More Emotions with Less Data Using Self-supervised Transfer Learning [0.0]
  We propose a novel transfer learning method for speech emotion recognition. With as low as 125 examples per emotion class, we were able to reach a higher accuracy than a strong baseline trained on 8 times more data.
  arXiv  Detail & Related papers  (2020-11-11T06:18:31Z)
• Continuous Emotion Recognition via Deep Convolutional Autoencoder and Support Vector Regressor [70.2226417364135]
  It is crucial that the machine should be able to recognize the emotional state of the user with high accuracy. Deep neural networks have been used with great success in recognizing emotions. We present a new model for continuous emotion recognition based on facial expression recognition.
  arXiv  Detail & Related papers  (2020-01-31T17:47:16Z)
• Facial Feedback for Reinforcement Learning: A Case Study and Offline Analysis Using the TAMER Framework [51.237191651923666]
  We investigate the potential of agent learning from trainers' facial expressions via interpreting them as evaluative feedback. With designed CNN-RNN model, our analysis shows that telling trainers to use facial expressions and competition can improve the accuracies for estimating positive and negative feedback. Our results with a simulation experiment show that learning solely from predicted feedback based on facial expressions is possible.
  arXiv  Detail & Related papers  (2020-01-23T17:50:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.