The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation

• URL: http://arxiv.org/abs/2604.26347v1
• Date: Wed, 29 Apr 2026 06:59:48 GMT
• Title: The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation
• Abstract summary: Objective metrics for emotional expressiveness are vital for speech generation, particularly in expressive synthesis and voice conversion.<n>We compute cosine similarity of embeddings from encoders like emotion2vec, assuming they capture affective cues despite linguistic and speaker variations.<n>Despite high classification accuracy, these latent spaces are unsuitable for zero-shot similarity evaluation.
• Score: 66.28943752785906
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Objective metrics for emotional expressiveness are vital for speech generation, particularly in expressive synthesis and voice conversion requiring emotional prosody transfer. To quantify this, the field widely relies on emotion similarity between reference and generated samples. This approach computes cosine similarity of embeddings from encoders like emotion2vec, assuming they capture affective cues despite linguistic and speaker variations. We challenge this assumption through controlled adversarial tasks and human alignment tests. Despite high classification accuracy, these latent spaces are unsuitable for zero-shot similarity evaluation. Representational limitations cause linguistic and speaker interference to overshadow emotional features, degrading discriminative ability. Consequently, the metric misaligns with human perception. This acoustic vulnerability reveals it rewards acoustic mimicry over genuine emotional synthesis.

Related papers

• On the Emotion Understanding of Synthesized Speech [63.13411068766772]
  Emotion is a core paralinguistic feature in voice interaction.<n>Current Speech Emotion Recognition (SER) models can not generalize to synthesized speech.<n>generative Speech Language Models (SLMs) tend to infer emotion from textual semantics while ignoring paralinguistic cues.
  arXiv  Detail & Related papers  (2026-03-17T13:11:14Z)
• Semantic Differentiation in Speech Emotion Recognition: Insights from Descriptive and Expressive Speech Roles [4.516156697420418]
  Speech Emotion Recognition (SER) is essential for improving human-computer interaction.<n>We distinguish between descriptive semantics, which represents the contextual content of speech, and expressive semantics, which reflects the speaker's emotional state.<n>Our findings inform SER applications in human-AI interaction and pave the way for more context-aware AI systems.
  arXiv  Detail & Related papers  (2025-10-03T14:42:35Z)
• DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speech [49.128847336227636]
  Cross-speaker emotion transfer in speech synthesis relies on extracting speaker-independent emotion embeddings for accurate emotion modeling.<n>We propose DiEmo-TTS, a self-supervised distillation method to minimize emotional information loss and preserve speaker identity.
  arXiv  Detail & Related papers  (2025-05-26T08:47:39Z)
• Improving speaker verification robustness with synthetic emotional utterances [14.63248006004598]
  A speaker verification (SV) system offers an authentication service designed to confirm whether a given speech sample originates from a specific speaker.<n>Previous models exhibit high error rates when dealing with emotional utterances compared to neutral ones.<n>This issue primarily stems from the limited availability of labeled emotional speech data.<n>We propose a novel approach employing the CycleGAN framework to serve as a data augmentation method.
  arXiv  Detail & Related papers  (2024-11-30T02:18:26Z)
• AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesis [13.918119853846838]
  Affect is an emotional characteristic encompassing valence, arousal, and intensity, and is a crucial attribute for enabling authentic conversations. We propose AffectEcho, an emotion translation model, that uses a Vector Quantized codebook to model emotions within a quantized space. We demonstrate the effectiveness of our approach in controlling the emotions of generated speech while preserving identity, style, and emotional cadence unique to each speaker.
  arXiv  Detail & Related papers  (2023-08-16T06:28:29Z)
• In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesis [15.16865739526702]
  We introduce a methodology that uses self-supervised networks to disentangle the lexical, speaker, and emotional content of the utterance. We then use a HiFiGAN vocoder to resynthesise the disentangled representations to a speech signal of the targeted emotion. Results reveal that the proposed approach is aptly conditioned on the emotional content of input speech and is capable of synthesising natural-sounding speech for a target emotion.
  arXiv  Detail & Related papers  (2023-06-02T21:02:51Z)
• Speech Synthesis with Mixed Emotions [77.05097999561298]
  We propose a novel formulation that measures the relative difference between the speech samples of different emotions. We then incorporate our formulation into a sequence-to-sequence emotional text-to-speech framework. At run-time, we control the model to produce the desired emotion mixture by manually defining an emotion attribute vector.
  arXiv  Detail & Related papers  (2022-08-11T15:45:58Z)
• Emotion Intensity and its Control for Emotional Voice Conversion [77.05097999561298]
  Emotional voice conversion (EVC) seeks to convert the emotional state of an utterance while preserving the linguistic content and speaker identity. In this paper, we aim to explicitly characterize and control the intensity of emotion. We propose to disentangle the speaker style from linguistic content and encode the speaker style into a style embedding in a continuous space that forms the prototype of emotion embedding.
  arXiv  Detail & Related papers  (2022-01-10T02:11:25Z)
• Textless Speech Emotion Conversion using Decomposed and Discrete Representations [49.55101900501656]
  We decompose speech into discrete and disentangled learned representations, consisting of content units, F0, speaker, and emotion. First, we modify the speech content by translating the content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder.
  arXiv  Detail & Related papers  (2021-11-14T18:16:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.