Revisiting Acoustic Similarity in Emotional Speech and Music via Self-Supervised Representations

• URL: http://arxiv.org/abs/2409.17899v1
• Date: Thu, 26 Sep 2024 14:49:09 GMT
• Title: Revisiting Acoustic Similarity in Emotional Speech and Music via Self-Supervised Representations
• Authors: Yujia Sun, Zeyu Zhao, Korin Richmond, Yuanchao Li,
• Abstract summary: Emotion recognition from speech and music shares similarities due to their acoustic overlap, which has led to interest in transferring knowledge between these domains. We revisit the acoustic similarity between emotion speech and music, starting with an analysis of the layerwise behavior of Self-Supervised Learning (SSL) models. Lastly, we explore the acoustic similarities between emotional speech and music using Frechet audio distance for individual emotions, uncovering the issue of emotion bias in both speech and music SSL models.
• Score: 14.058054715262275
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Emotion recognition from speech and music shares similarities due to their acoustic overlap, which has led to interest in transferring knowledge between these domains. However, the shared acoustic cues between speech and music, particularly those encoded by Self-Supervised Learning (SSL) models, remain largely unexplored, given the fact that SSL models for speech and music have rarely been applied in cross-domain research. In this work, we revisit the acoustic similarity between emotion speech and music, starting with an analysis of the layerwise behavior of SSL models for Speech Emotion Recognition (SER) and Music Emotion Recognition (MER). Furthermore, we perform cross-domain adaptation by comparing several approaches in a two-stage fine-tuning process, examining effective ways to utilize music for SER and speech for MER. Lastly, we explore the acoustic similarities between emotional speech and music using Frechet audio distance for individual emotions, uncovering the issue of emotion bias in both speech and music SSL models. Our findings reveal that while speech and music SSL models do capture shared acoustic features, their behaviors can vary depending on different emotions due to their training strategies and domain-specificities. Additionally, parameter-efficient fine-tuning can enhance SER and MER performance by leveraging knowledge from each other. This study provides new insights into the acoustic similarity between emotional speech and music, and highlights the potential for cross-domain generalization to improve SER and MER systems.

Related papers

• Rethinking Emotion Bias in Music via Frechet Audio Distance [11.89773040110695]
  We conduct a study on Music Emotion Recognition (MER) and Emotional Music Generation (EMG) We employ diverse audio encoders alongside the Frechet Audio Distance (FAD), a reference-free evaluation metric.
  arXiv  Detail & Related papers  (2024-09-23T20:59:15Z)
• Joint Learning of Emotions in Music and Generalized Sounds [6.854732863866882]
  We propose the use of multiple datasets as a multi-domain learning technique. Our approach involves creating a common space encompassing features that characterize both generalized sounds and music. We performed joint learning on the common feature space, leveraging heterogeneous model architectures.
  arXiv  Detail & Related papers  (2024-08-04T12:19:03Z)
• Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion [81.1492897350032]
  Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. We propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion.
  arXiv  Detail & Related papers  (2023-12-29T08:06:45Z)
• Prompting Audios Using Acoustic Properties For Emotion Representation [36.275219004598874]
  We propose the use of natural language descriptions (or prompts) to better represent emotions. We use acoustic properties that are correlated to emotion like pitch, intensity, speech rate, and articulation rate to automatically generate prompts. Our results show that the acoustic prompts significantly improve the model's performance in various Precision@K metrics.
  arXiv  Detail & Related papers  (2023-10-03T13:06:58Z)
• Empirical Interpretation of the Relationship Between Speech Acoustic Context and Emotion Recognition [28.114873457383354]
  Speech emotion recognition (SER) is vital for obtaining emotional intelligence and understanding the contextual meaning of speech. In practice, speech emotions are treated as single labels over an acoustic segment for a given time duration. This research explores the implication of acoustic context and phone boundaries on local markers for SER using an attention-based approach.
  arXiv  Detail & Related papers  (2023-06-30T09:21:48Z)
• Emotion Intensity and its Control for Emotional Voice Conversion [77.05097999561298]
  Emotional voice conversion (EVC) seeks to convert the emotional state of an utterance while preserving the linguistic content and speaker identity. In this paper, we aim to explicitly characterize and control the intensity of emotion. We propose to disentangle the speaker style from linguistic content and encode the speaker style into a style embedding in a continuous space that forms the prototype of emotion embedding.
  arXiv  Detail & Related papers  (2022-01-10T02:11:25Z)
• Reinforcement Learning for Emotional Text-to-Speech Synthesis with Improved Emotion Discriminability [82.39099867188547]
  Emotional text-to-speech synthesis (ETTS) has seen much progress in recent years. We propose a new interactive training paradigm for ETTS, denoted as i-ETTS. We formulate an iterative training strategy with reinforcement learning to ensure the quality of i-ETTS optimization.
  arXiv  Detail & Related papers  (2021-04-03T13:52:47Z)
• Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset [84.53659233967225]
  Emotional voice conversion aims to transform emotional prosody in speech while preserving the linguistic content and speaker identity. We propose a novel framework based on variational auto-encoding Wasserstein generative adversarial network (VAW-GAN) We show that the proposed framework achieves remarkable performance by consistently outperforming the baseline framework.
  arXiv  Detail & Related papers  (2020-10-28T07:16:18Z)
• Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space [80.49156615923106]
  Matching images and music with similar emotions might help to make emotion perceptions more vivid and stronger. Existing emotion-based image and music matching methods either employ limited categorical emotion states or train the matching model using an impractical multi-stage pipeline. In this paper, we study end-to-end matching between image and music based on emotions in the continuous valence-arousal (VA) space.
  arXiv  Detail & Related papers  (2020-08-22T20:12:23Z)
• Music Gesture for Visual Sound Separation [121.36275456396075]
  "Music Gesture" is a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals.
  arXiv  Detail & Related papers  (2020-04-20T17:53:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.