SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training

• URL: http://arxiv.org/abs/2609.02941v1
• Date: Mon, 31 Aug 2026 16:48:43 GMT
• Title: SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training
• Abstract summary: Speech emotion recognition (SER) faces two fundamental challenges: scarcity of labeled data and inter-speaker variability.<n>We propose SISER (Speaker-Invariant Speech Emotion Recognition), integrating wav2vec 2.0 as a feature encoder and ECAPA-TDNN as a speaker discriminator.<n>SISER achieves a UA of 60.63%, outperforming the baseline (51.15%) and wav2vec 2.0 without speaker suppression (56.46%)
• Score: 4.0913617519821255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speech emotion recognition (SER) faces two fundamental challenges: scarcity of labeled data and inter-speaker variability, both of which hinder generalization of emotion recognition systems. While prior adversarial approaches address speaker variability, they fall short in leveraging powerful pre-trained representations. We propose SISER (Speaker-Invariant Speech Emotion Recognition), integrating wav2vec 2.0 as a feature encoder and ECAPA-TDNN as a speaker discriminator within an entropy-based adversarial training scheme. wav2vec 2.0 provides rich self-supervised representations that alleviate dependency on large labeled datasets, while ECAPA-TDNN enables suppression of speaker identity via a stronger adversarial signal than shallow classifiers. Evaluated on IEMOCAP, SISER achieves a UA of 60.63%, outperforming the baseline (51.15%) and wav2vec 2.0 without speaker suppression (56.46%), with ablation emphasizing that the choice of speaker classifier architecture is a key factor.

Related papers

• Dual-Scale State-Space Modeling with Speaker-Wise Dynamic CRF for Speech Emotion Recognition in Conversation [1.6389263804357965]
  We propose DSSM-CRF, an audio-only architecture for emotion recognition.<n>Bidirectional state-space models encode fused self-supervised speech representations at frame and dialogue scales.<n> Consecutive utterances in a speaker's chain form a transition pair whose score combines a corpus-level transition matrix with a residual predicted from the two contextualized utterances.
  arXiv  Detail & Related papers  (2026-08-23T12:56:03Z)
• Towards Robust Overlapping Speech Detection: A Speaker-Aware Progressive Approach Using WavLM [53.17360668423001]
  Overlapping Speech Detection (OSD) aims to identify regions where multiple speakers overlap in a conversation.<n>This work proposes a speaker-aware progressive OSD model that leverages a progressive training strategy to enhance the correlation between subtasks.<n> Experimental results show that the proposed method achieves state-of-the-art performance, with an F1 score of 82.76% on the AMI test set.
  arXiv  Detail & Related papers  (2025-05-29T07:47:48Z)
• Use of Speech Impairment Severity for Dysarthric Speech Recognition [37.93801885333925]
  This paper proposes a novel set of techniques to use both severity and speaker-identity in dysarthric speech recognition. Experiments conducted on UASpeech suggest incorporating speech impairment severity into state-of-the-art hybrid DNN, E2E Conformer and pre-trained Wav2vec 2.0 ASR systems.
  arXiv  Detail & Related papers  (2023-05-18T02:42:59Z)
• Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition [57.31233839489528]
  This paper explores approaches to integrate domain adapted SSL pre-trained models into TDNN and Conformer ASR systems for dysarthric and elderly speech recognition.
  arXiv  Detail & Related papers  (2023-02-28T13:39:17Z)
• Acoustic-to-articulatory Inversion based on Speech Decomposition and Auxiliary Feature [7.363994037183394]
  We pre-train a speech decomposition network to decompose audio speech into speaker embedding and content embedding. We then propose a novel auxiliary feature network to estimate the lip auxiliary features from the personalized speech features. Experimental results show that, compared with the state-of-the-art only using the audio speech feature, the proposed method reduces the average RMSE by 0.25 and increases the average correlation coefficient by 2.0%.
  arXiv  Detail & Related papers  (2022-04-02T14:47:19Z)
• On-the-Fly Feature Based Rapid Speaker Adaptation for Dysarthric and Elderly Speech Recognition [53.17176024917725]
  Scarcity of speaker-level data limits the practical use of data-intensive model based speaker adaptation methods. This paper proposes two novel forms of data-efficient, feature-based on-the-fly speaker adaptation methods.
  arXiv  Detail & Related papers  (2022-03-28T09:12:24Z)
• Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition [57.15942628305797]
  Articulatory features are invariant to acoustic signal distortion and have been successfully incorporated into automatic speech recognition systems for normal speech. This paper presents a cross-domain acoustic-to-articulatory (A2A) inversion approach that utilizes the parallel acoustic-articulatory data of the 15-hour TORGO corpus in model training. Cross-domain adapted to the 102.7-hour UASpeech corpus and to produce articulatory features.
  arXiv  Detail & Related papers  (2022-03-19T08:47:18Z)
• Fine-tuning wav2vec2 for speaker recognition [3.69563307866315]
  We study the effectiveness of the pre-trained weights on the speaker recognition task, and how to pool the wav2vec2 output sequence into a fixed-length speaker embedding. To adapt the framework to speaker recognition, we propose a single-utterance classification variant with CE or AAM softmax loss, and an utterance-pair classification variant with BCE loss.
  arXiv  Detail & Related papers  (2021-09-30T12:16:47Z)
• An Attribute-Aligned Strategy for Learning Speech Representation [57.891727280493015]
  We propose an attribute-aligned learning strategy to derive speech representation that can flexibly address these issues by attribute-selection mechanism. Specifically, we propose a layered-representation variational autoencoder (LR-VAE), which factorizes speech representation into attribute-sensitive nodes. Our proposed method achieves competitive performances on identity-free SER and a better performance on emotionless SV.
  arXiv  Detail & Related papers  (2021-06-05T06:19:14Z)
• Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement [102.48582597586233]
  We present a U-Net based attention model, U-Net$_At$, to enhance adversarial speech signals. We conduct experiments on the automatic speech recognition (ASR) task with adversarial audio attacks.
  arXiv  Detail & Related papers  (2020-03-31T02:16:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.