Speech Emotion Recognition via an Attentive Time-Frequency Neural Network

• URL: http://arxiv.org/abs/2210.12430v1
• Date: Sat, 22 Oct 2022 12:18:26 GMT
• Title: Speech Emotion Recognition via an Attentive Time-Frequency Neural Network
• Abstract summary: We propose an attentive time-frequency neural network (ATFNN) for speech emotion recognition (SER) Specifically, we design a TFNN with a frequency-domain encoder (F-Encoder) based on the Transformer encoder and a time-domain encoder (T-Encoder) based on the Bidirectional Long Short-Term Memory (Bi-LSTM) To handle the second issue, we also adopt time-frequency attention with a frequency-attention network (F-Attention) and a time-attention network (T-Attention) to focus on the emotion-related frequency band ranges and time frame ranges.
• Score: 21.87456615530486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spectrogram is commonly used as the input feature of deep neural networks to learn the high(er)-level time-frequency pattern of speech signal for speech emotion recognition (SER). \textcolor{black}{Generally, different emotions correspond to specific energy activations both within frequency bands and time frames on spectrogram, which indicates the frequency and time domains are both essential to represent the emotion for SER. However, recent spectrogram-based works mainly focus on modeling the long-term dependency in time domain, leading to these methods encountering the following two issues: (1) neglecting to model the emotion-related correlations within frequency domain during the time-frequency joint learning; (2) ignoring to capture the specific frequency bands associated with emotions.} To cope with the issues, we propose an attentive time-frequency neural network (ATFNN) for SER, including a time-frequency neural network (TFNN) and time-frequency attention. Specifically, aiming at the first issue, we design a TFNN with a frequency-domain encoder (F-Encoder) based on the Transformer encoder and a time-domain encoder (T-Encoder) based on the Bidirectional Long Short-Term Memory (Bi-LSTM). The F-Encoder and T-Encoder model the correlations within frequency bands and time frames, respectively, and they are embedded into a time-frequency joint learning strategy to obtain the time-frequency patterns for speech emotions. Moreover, to handle the second issue, we also adopt time-frequency attention with a frequency-attention network (F-Attention) and a time-attention network (T-Attention) to focus on the emotion-related frequency band ranges and time frame ranges, which can enhance the discriminability of speech emotion features.

Related papers

• STARK: Spatio-Temporal Attention for Representation of Keypoints for Continuous Sign Language Recognition [0.0]
  This paper introduces a unified-temporal attention network that computes attention scores both spatially (across keypoints) and temporally (within local windows)<n>The encoder contains approximately $70-80$ fewer parameters than existing state-of-the-art models.
  arXiv  Detail & Related papers  (2026-03-17T06:29:57Z)
• Neuromorphic Wireless Split Computing with Resonate-and-Fire Neurons [69.73249913506042]
  This paper investigates a wireless split computing architecture that employs resonate-and-fire (RF) neurons to process time-domain signals directly.<n>By resonating at tunable frequencies, RF neurons extract time-localized spectral features while maintaining low spiking activity.<n> Experimental results show that the proposed RF-SNN architecture achieves comparable accuracy to conventional LIF-SNNs and ANNs.
  arXiv  Detail & Related papers  (2025-06-24T21:14:59Z)
• SigWavNet: Learning Multiresolution Signal Wavelet Network for Speech Emotion Recognition [17.568724398229232]
  Speech emotion recognition (SER) plays an important role in emotional states from deciphering speech signals. This paper introduces a new end-to-end (E2E) deep learning multi-resolution framework for SER. It exploits the capabilities of wavelets for effective localization in both time and frequency domains.
  arXiv  Detail & Related papers  (2025-02-01T04:18:06Z)
• NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding [60.74434735079253]
  We propose a method known as Neighborhood Attention with Convolutions TCN (NAC-TCN) We accomplish this by introducing a causal version of Dilated Neighborhood Attention while incorporating it with convolutions. Our model achieves comparable, better, or state-of-the-art performance over TCNs, TCAN, LSTMs, and GRUs while requiring fewer parameters on standard emotion recognition datasets.
  arXiv  Detail & Related papers  (2023-12-12T18:41:30Z)
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis [1.4277428617774877]
  We present Vocos, a new model that directly generates Fourier spectral coefficients. It substantially improves computational efficiency, achieving an order of magnitude increase in speed compared to prevailing time-domain neural vocoding approaches.
  arXiv  Detail & Related papers  (2023-06-01T15:40:32Z)
• Time-space-frequency feature Fusion for 3-channel motor imagery classification [0.0]
  This study introduces TSFF-Net, a novel network architecture that integrates time-space-frequency features. TSFF-Net comprises four main components: time-frequency representation, time-frequency feature extraction, time-space feature extraction, and feature fusion and classification. Experimental results demonstrate that TSFF-Net not only compensates for the shortcomings of single-mode feature extraction networks in EEG decoding, but also outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-04-04T02:01:48Z)
• Braille Letter Reading: A Benchmark for Spatio-Temporal Pattern Recognition on Neuromorphic Hardware [50.380319968947035]
  Recent deep learning approaches have reached accuracy in such tasks, but their implementation on conventional embedded solutions is still computationally very and energy expensive. We propose a new benchmark for computing tactile pattern recognition at the edge through letters reading. We trained and compared feed-forward and recurrent spiking neural networks (SNNs) offline using back-propagation through time with surrogate gradients, then we deployed them on the Intel Loihimorphic chip for efficient inference. Our results show that the LSTM outperforms the recurrent SNN in terms of accuracy by 14%. However, the recurrent SNN on Loihi is 237 times more energy
  arXiv  Detail & Related papers  (2022-05-30T14:30:45Z)
• Adaptive Frequency Learning in Two-branch Face Forgery Detection [66.91715092251258]
  We propose Adaptively learn Frequency information in the two-branch Detection framework, dubbed AFD. We liberate our network from the fixed frequency transforms, and achieve better performance with our data- and task-dependent transform layers.
  arXiv  Detail & Related papers  (2022-03-27T14:25:52Z)
• Time and Frequency Network for Human Action Detection in Videos [6.78349879472022]
  We propose an end-to-end network that considers the time and frequency features simultaneously, named TFNet. To obtain the action patterns, these two features are deeply fused under the attention mechanism.
  arXiv  Detail & Related papers  (2021-03-08T11:42:05Z)
• Non-linear frequency warping using constant-Q transformation for speech emotion recognition [10.560561286520045]
  We explore the constant-Q transform (CQT) for speech emotion recognition (SER) CQT-based time-frequency analysis provides variable spectro-temporal resolution with higher frequency resolution at lower frequencies. We present a comparative analysis of short-term acoustic features based on STFT and CQT for SER with deep neural network (DNN) as a back-end classifier.
  arXiv  Detail & Related papers  (2021-02-08T06:57:16Z)
• Attention and Encoder-Decoder based models for transforming articulatory movements at different speaking rates [60.02121449986413]
  We propose an encoder-decoder architecture using LSTMs which generates smoother predicted articulatory trajectories. We analyze amplitude of the transformed articulatory movements at different rates compared to their original counterparts. We observe that AstNet could model both duration and extent of articulatory movements better than the existing transformation techniques.
  arXiv  Detail & Related papers  (2020-06-04T19:33:26Z)
• Multi-Time-Scale Convolution for Emotion Recognition from Speech Audio Signals [7.219077740523682]
  We introduce the multi-time-scale (MTS) method to create flexibility towards temporal variations when analyzing audio data. We evaluate MTS and standard convolutional layers in different architectures for emotion recognition from speech audio, using 4 datasets of different sizes.
  arXiv  Detail & Related papers  (2020-03-06T12:28:04Z)
• An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos [64.91614454412257]
  We propose to recognize video emotions in an end-to-end manner based on convolutional neural networks (CNNs) Specifically, we develop a deep Visual-Audio Attention Network (VAANet), a novel architecture that integrates spatial, channel-wise, and temporal attentions into a visual 3D CNN and temporal attentions into an audio 2D CNN.
  arXiv  Detail & Related papers  (2020-02-12T15:33:59Z)
• WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss [74.11899135025503]
  Tacotron-based text-to-speech (TTS) systems directly synthesize speech from text input. We propose a new training scheme for Tacotron-based TTS, referred to as WaveTTS, that has 2 loss functions. WaveTTS ensures both the quality of the acoustic features and the resulting speech waveform.
  arXiv  Detail & Related papers  (2020-02-02T15:51:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.