How you feelin'? Learning Emotions and Mental States in Movie Scenes

• URL: http://arxiv.org/abs/2304.05634v1
• Date: Wed, 12 Apr 2023 06:31:14 GMT
• Title: How you feelin'? Learning Emotions and Mental States in Movie Scenes
• Authors: Dhruv Srivastava and Aditya Kumar Singh and Makarand Tapaswi
• Abstract summary: We formulate emotion understanding as predicting a diverse and multi-label set of emotions at the level of a movie scene. EmoTx is a multimodal Transformer-based architecture that ingests videos, multiple characters, and dialog utterances to make joint predictions.
• Score: 9.368590075496149
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Movie story analysis requires understanding characters' emotions and mental states. Towards this goal, we formulate emotion understanding as predicting a diverse and multi-label set of emotions at the level of a movie scene and for each character. We propose EmoTx, a multimodal Transformer-based architecture that ingests videos, multiple characters, and dialog utterances to make joint predictions. By leveraging annotations from the MovieGraphs dataset, we aim to predict classic emotions (e.g. happy, angry) and other mental states (e.g. honest, helpful). We conduct experiments on the most frequently occurring 10 and 25 labels, and a mapping that clusters 181 labels to 26. Ablation studies and comparison against adapted state-of-the-art emotion recognition approaches shows the effectiveness of EmoTx. Analyzing EmoTx's self-attention scores reveals that expressive emotions often look at character tokens while other mental states rely on video and dialog cues.

Related papers

• Language Models (Mostly) Do Not Consider Emotion Triggers When Predicting Emotion [87.18073195745914]
  We investigate how well human-annotated emotion triggers correlate with features deemed salient in their prediction of emotions. Using EmoTrigger, we evaluate the ability of large language models to identify emotion triggers. Our analysis reveals that emotion triggers are largely not considered salient features for emotion prediction models, instead there is intricate interplay between various features and the task of emotion detection.
  arXiv  Detail & Related papers  (2023-11-16T06:20:13Z)
• Why Do You Feel This Way? Summarizing Triggers of Emotions in Social Media Posts [61.723046082145416]
  We introduce CovidET (Emotions and their Triggers during Covid-19), a dataset of 1,900 English Reddit posts related to COVID-19. We develop strong baselines to jointly detect emotions and summarize emotion triggers. Our analyses show that CovidET presents new challenges in emotion-specific summarization, as well as multi-emotion detection in long social media posts.
  arXiv  Detail & Related papers  (2022-10-22T19:10:26Z)
• Experiencer-Specific Emotion and Appraisal Prediction [13.324006587838523]
  Emotion classification in NLP assigns emotions to texts, such as sentences or paragraphs. We focus on the experiencers of events, and assign an emotion (if any holds) to each of them. Our experiencer-aware models of emotions and appraisals outperform the experiencer-agnostic baselines.
  arXiv  Detail & Related papers  (2022-10-21T16:04:27Z)
• Speech Synthesis with Mixed Emotions [77.05097999561298]
  We propose a novel formulation that measures the relative difference between the speech samples of different emotions. We then incorporate our formulation into a sequence-to-sequence emotional text-to-speech framework. At run-time, we control the model to produce the desired emotion mixture by manually defining an emotion attribute vector.
  arXiv  Detail & Related papers  (2022-08-11T15:45:58Z)
• MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild [56.61912265155151]
  We propose MAFW, a large-scale compound affective database with 10,045 video-audio clips in the wild. Each clip is annotated with a compound emotional category and a couple of sentences that describe the subjects' affective behaviors in the clip. For the compound emotion annotation, each clip is categorized into one or more of the 11 widely-used emotions, i.e., anger, disgust, fear, happiness, neutral, sadness, surprise, contempt, anxiety, helplessness, and disappointment.
  arXiv  Detail & Related papers  (2022-08-01T13:34:33Z)
• x-enVENT: A Corpus of Event Descriptions with Experiencer-specific Emotion and Appraisal Annotations [13.324006587838523]
  We argue that a classification setup for emotion analysis should be performed in an integrated manner, including the different semantic roles that participate in an emotion episode. Based on appraisal theories in psychology, we compile an English corpus of written event descriptions. The descriptions depict emotion-eliciting circumstances, and they contain mentions of people who responded emotionally.
  arXiv  Detail & Related papers  (2022-03-21T12:02:06Z)
• Emotion Intensity and its Control for Emotional Voice Conversion [77.05097999561298]
  Emotional voice conversion (EVC) seeks to convert the emotional state of an utterance while preserving the linguistic content and speaker identity. In this paper, we aim to explicitly characterize and control the intensity of emotion. We propose to disentangle the speaker style from linguistic content and encode the speaker style into a style embedding in a continuous space that forms the prototype of emotion embedding.
  arXiv  Detail & Related papers  (2022-01-10T02:11:25Z)
• A Circular-Structured Representation for Visual Emotion Distribution Learning [82.89776298753661]
  We propose a well-grounded circular-structured representation to utilize the prior knowledge for visual emotion distribution learning. To be specific, we first construct an Emotion Circle to unify any emotional state within it. On the proposed Emotion Circle, each emotion distribution is represented with an emotion vector, which is defined with three attributes.
  arXiv  Detail & Related papers  (2021-06-23T14:53:27Z)
• Emotion Dynamics in Movie Dialogues [25.289525325790414]
  We introduce a framework to track emotion dynamics through one's utterances. We use this approach to trace emotional arcs of movie characters. We analyze thousands of such character arcs to test hypotheses that inform our broader understanding of stories.
  arXiv  Detail & Related papers  (2021-03-01T23:02:16Z)
• Fine-grained Emotion and Intent Learning in Movie Dialogues [1.2891210250935146]
  We propose a novel large-scale emotional dialogue dataset, consisting of 1M dialogues retrieved from the OpenSubtitles corpus. This work explains the complex pipeline used to preprocess movie subtitles and select good movie dialogues to annotate. This scale of emotional dialogue classification has never been attempted before, both in terms of dataset size and fine-grained emotion and intent categories.
  arXiv  Detail & Related papers  (2020-12-25T20:29:56Z)
• PO-EMO: Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry [26.172030802168752]
  We consider emotions in poetry as they are elicited in the reader, rather than what is expressed in the text or intended by the author. We conceptualize a set of aesthetic emotions that are predictive of aesthetic appreciation in the reader, and allow the annotation of multiple labels per line to capture mixed emotions within their context.
  arXiv  Detail & Related papers  (2020-03-17T13:54:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.