Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition

• URL: http://arxiv.org/abs/2009.09629v3
• Date: Wed, 7 Oct 2020 05:09:31 GMT
• Title: Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition
• Authors: Wenliang Dai, Zihan Liu, Tiezheng Yu and Pascale Fung
• Abstract summary: We propose a modality-transferable model with emotion embeddings to tackle the aforementioned issues. Our model achieves state-of-the-art performance on most of the emotion categories. Our model also outperforms existing baselines in the zero-shot and few-shot scenarios for unseen emotions.
• Score: 55.44502358463217
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the recent achievements made in the multi-modal emotion recognition task, two problems still exist and have not been well investigated: 1) the relationship between different emotion categories are not utilized, which leads to sub-optimal performance; and 2) current models fail to cope well with low-resource emotions, especially for unseen emotions. In this paper, we propose a modality-transferable model with emotion embeddings to tackle the aforementioned issues. We use pre-trained word embeddings to represent emotion categories for textual data. Then, two mapping functions are learned to transfer these embeddings into visual and acoustic spaces. For each modality, the model calculates the representation distance between the input sequence and target emotions and makes predictions based on the distances. By doing so, our model can directly adapt to the unseen emotions in any modality since we have their pre-trained embeddings and modality mapping functions. Experiments show that our model achieves state-of-the-art performance on most of the emotion categories. In addition, our model also outperforms existing baselines in the zero-shot and few-shot scenarios for unseen emotions.

Related papers

• Language Models (Mostly) Do Not Consider Emotion Triggers When Predicting Emotion [87.18073195745914]
  We investigate how well human-annotated emotion triggers correlate with features deemed salient in their prediction of emotions. Using EmoTrigger, we evaluate the ability of large language models to identify emotion triggers. Our analysis reveals that emotion triggers are largely not considered salient features for emotion prediction models, instead there is intricate interplay between various features and the task of emotion detection.
  arXiv  Detail & Related papers  (2023-11-16T06:20:13Z)
• High-Level Context Representation for Emotion Recognition in Images [4.987022981158291]
  We propose an approach for high-level context representation extraction from images. The model relies on a single cue and a single encoding stream to correlate this representation with emotions. Our approach is more efficient than previous models and can be easily deployed to address real-world problems related to emotion recognition.
  arXiv  Detail & Related papers  (2023-05-05T13:20:41Z)
• Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats [0.0]
  We show how we can build a single model that can transition between different configurations. We show that Demux can simultaneously transfer knowledge in a zero-shot manner to a new language.
  arXiv  Detail & Related papers  (2022-10-31T22:32:36Z)
• Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies [106.62835060095532]
  We discuss several key aspects of multi-modal emotion recognition (MER) We begin with a brief introduction on widely used emotion representation models and affective modalities. We then summarize existing emotion annotation strategies and corresponding computational tasks. Finally, we outline several real-world applications and discuss some future directions.
  arXiv  Detail & Related papers  (2021-08-18T21:55:20Z)
• Fine-Grained Emotion Prediction by Modeling Emotion Definitions [26.098917459551167]
  We propose a new framework for fine-grained emotion prediction in the text through emotion definition modeling. Our models outperform existing state-of-the-art for fine-grained emotion dataset GoEmotions.
  arXiv  Detail & Related papers  (2021-07-26T12:11:18Z)
• A Circular-Structured Representation for Visual Emotion Distribution Learning [82.89776298753661]
  We propose a well-grounded circular-structured representation to utilize the prior knowledge for visual emotion distribution learning. To be specific, we first construct an Emotion Circle to unify any emotional state within it. On the proposed Emotion Circle, each emotion distribution is represented with an emotion vector, which is defined with three attributes.
  arXiv  Detail & Related papers  (2021-06-23T14:53:27Z)
• Enhancing Cognitive Models of Emotions with Representation Learning [58.2386408470585]
  We present a novel deep learning-based framework to generate embedding representations of fine-grained emotions. Our framework integrates a contextualized embedding encoder with a multi-head probing model. Our model is evaluated on the Empathetic Dialogue dataset and shows the state-of-the-art result for classifying 32 emotions.
  arXiv  Detail & Related papers  (2021-04-20T16:55:15Z)
• Facial Expression Editing with Continuous Emotion Labels [76.36392210528105]
  Deep generative models have achieved impressive results in the field of automated facial expression editing. We propose a model that can be used to manipulate facial expressions in facial images according to continuous two-dimensional emotion labels.
  arXiv  Detail & Related papers  (2020-06-22T13:03:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.