Challenges in Emotion Style Transfer: An Exploration with a Lexical Substitution Pipeline

• URL: http://arxiv.org/abs/2005.07617v1
• Date: Fri, 15 May 2020 16:11:33 GMT
• Title: Challenges in Emotion Style Transfer: An Exploration with a Lexical Substitution Pipeline
• Authors: David Helbig and Enrica Troiano and Roman Klinger
• Abstract summary: We design a transparent emotion style transfer pipeline based on three steps. We explore what cases lexical substitution can vary the emotional load of texts. We find, indeed, that simultaneous adjustments of content and emotion are conflicting objectives.
• Score: 16.3589458084367
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We propose the task of emotion style transfer, which is particularly challenging, as emotions (here: anger, disgust, fear, joy, sadness, surprise) are on the fence between content and style. To understand the particular difficulties of this task, we design a transparent emotion style transfer pipeline based on three steps: (1) select the words that are promising to be substituted to change the emotion (with a brute-force approach and selection based on the attention mechanism of an emotion classifier), (2) find sets of words as candidates for substituting the words (based on lexical and distributional semantics), and (3) select the most promising combination of substitutions with an objective function which consists of components for content (based on BERT sentence embeddings), emotion (based on an emotion classifier), and fluency (based on a neural language model). This comparably straight-forward setup enables us to explore the task and understand in what cases lexical substitution can vary the emotional load of texts, how changes in content and style interact and if they are at odds. We further evaluate our pipeline quantitatively in an automated and an annotation study based on Tweets and find, indeed, that simultaneous adjustments of content and emotion are conflicting objectives: as we show in a qualitative analysis motivated by Scherer's emotion component model, this is particularly the case for implicit emotion expressions based on cognitive appraisal or descriptions of bodily reactions.

Related papers

• Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion [81.1492897350032]
  Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. We propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion.
  arXiv  Detail & Related papers  (2023-12-29T08:06:45Z)
• Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling [50.99252242917458]
  Conversational Speech Synthesis (CSS) aims to accurately express an utterance with the appropriate prosody and emotional inflection within a conversational setting. To address the issue of data scarcity, we meticulously create emotional labels in terms of category and intensity. Our model outperforms the baseline models in understanding and rendering emotions.
  arXiv  Detail & Related papers  (2023-12-19T08:47:50Z)
• Dynamic Causal Disentanglement Model for Dialogue Emotion Detection [77.96255121683011]
  We propose a Dynamic Causal Disentanglement Model based on hidden variable separation. This model effectively decomposes the content of dialogues and investigates the temporal accumulation of emotions. Specifically, we propose a dynamic temporal disentanglement model to infer the propagation of utterances and hidden variables.
  arXiv  Detail & Related papers  (2023-09-13T12:58:09Z)
• AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesis [13.918119853846838]
  Affect is an emotional characteristic encompassing valence, arousal, and intensity, and is a crucial attribute for enabling authentic conversations. We propose AffectEcho, an emotion translation model, that uses a Vector Quantized codebook to model emotions within a quantized space. We demonstrate the effectiveness of our approach in controlling the emotions of generated speech while preserving identity, style, and emotional cadence unique to each speaker.
  arXiv  Detail & Related papers  (2023-08-16T06:28:29Z)
• Emotion and Sentiment Guided Paraphrasing [3.5027291542274366]
  We introduce a new task of fine-grained emotional paraphrasing along emotion gradients. We reconstruct several widely used paraphrasing datasets by augmenting the input and target texts with their fine-grained emotion labels. We propose a framework for emotion and sentiment guided paraphrasing by leveraging pre-trained language models for conditioned text generation.
  arXiv  Detail & Related papers  (2023-06-08T20:59:40Z)
• Fine-Grained Emotional Paraphrasing along Emotion Gradients [0.0]
  We introduce a new task of fine-grained emotional paraphrasing along emotion gradients. We propose a framework for addressing this task by fine-tuning text-to-text Transformers through multi-task training.
  arXiv  Detail & Related papers  (2022-10-30T05:38:22Z)
• Speech Synthesis with Mixed Emotions [77.05097999561298]
  We propose a novel formulation that measures the relative difference between the speech samples of different emotions. We then incorporate our formulation into a sequence-to-sequence emotional text-to-speech framework. At run-time, we control the model to produce the desired emotion mixture by manually defining an emotion attribute vector.
  arXiv  Detail & Related papers  (2022-08-11T15:45:58Z)
• Emotion Intensity and its Control for Emotional Voice Conversion [77.05097999561298]
  Emotional voice conversion (EVC) seeks to convert the emotional state of an utterance while preserving the linguistic content and speaker identity. In this paper, we aim to explicitly characterize and control the intensity of emotion. We propose to disentangle the speaker style from linguistic content and encode the speaker style into a style embedding in a continuous space that forms the prototype of emotion embedding.
  arXiv  Detail & Related papers  (2022-01-10T02:11:25Z)
• Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies [106.62835060095532]
  We discuss several key aspects of multi-modal emotion recognition (MER) We begin with a brief introduction on widely used emotion representation models and affective modalities. We then summarize existing emotion annotation strategies and corresponding computational tasks. Finally, we outline several real-world applications and discuss some future directions.
  arXiv  Detail & Related papers  (2021-08-18T21:55:20Z)
• Multi-Task Learning and Adapted Knowledge Models for Emotion-Cause Extraction [18.68808042388714]
  We present solutions that tackle both emotion recognition and emotion cause detection in a joint fashion. Considering that common-sense knowledge plays an important role in understanding implicitly expressed emotions, we propose novel methods. We show performance improvement on both tasks when including common-sense reasoning and a multitask framework.
  arXiv  Detail & Related papers  (2021-06-17T20:11:04Z)
• EmoDNN: Understanding emotions from short texts through a deep neural network ensemble [2.459874436804819]
  We propose a framework that infers latent individual aspects from brief contents. We also present a novel ensemble classifier equipped with dynamic dropout convnets to extract emotions from textual context. Our proposed model can achieve a higher performance in recognizing emotion from noisy contents.
  arXiv  Detail & Related papers  (2021-06-03T09:17:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.