Dynamic Time-Alignment of Dimensional Annotations of Emotion using Recurrent Neural Networks

• URL: http://arxiv.org/abs/2209.10223v1
• Date: Wed, 21 Sep 2022 09:38:57 GMT
• Title: Dynamic Time-Alignment of Dimensional Annotations of Emotion using Recurrent Neural Networks
• Abstract summary: We propose a method that can compensate inconsistencies across annotations and synchronise the traces with the corresponding acoustic features. Results show that our method can significantly increase inter-annotator agreement, as well as correlation between traces and audio features.
• Score: 9.02236667251654
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Most automatic emotion recognition systems exploit time-continuous annotations of emotion to provide fine-grained descriptions of spontaneous expressions as observed in real-life interactions. As emotion is rather subjective, its annotation is usually performed by several annotators who provide a trace for a given dimension, i.e. a time-continuous series describing a dimension such as arousal or valence. However, annotations of the same expression are rarely consistent between annotators, either in time or in value, which adds bias and delay in the trace that is used to learn predictive models of emotion. We therefore propose a method that can dynamically compensate inconsistencies across annotations and synchronise the traces with the corresponding acoustic features using Recurrent Neural Networks. Experimental evaluations were carried on several emotion data sets that include Chinese, French, German, and Hungarian participants who interacted remotely in either noise-free conditions or in-the-wild. The results show that our method can significantly increase inter-annotator agreement, as well as correlation between traces and audio features, for both arousal and valence. In addition, improvements are obtained in the automatic prediction of these dimensions using simple light-weight models, especially for valence in noise-free conditions, and arousal for recordings captured in-the-wild.

Related papers

• Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue [44.39196719594625]
  We study 27 human-human interviews conducted in virtual reality.<n>We compare 20 time-series models while excluding lexical and acoustic information, and then systematically vary temporal context, interactional source, modality availability, and event timing.<n>Across grouped 10-fold cross-validation, the strongest probes reach up to.75 mean held-out AUROC from speaker-side behavior.
  arXiv  Detail & Related papers  (2026-09-14T22:07:34Z)
• Decoding Naturalistic Emotion Dynamics from the Brain: An LLM-Enhanced Regression Framework [3.8543064191088203]
  Decoding emotional states from neural signals has been typically framed as a discrete, single-label classification task.<n>This study reconceptualizes emotion decoding by adopting a multi-target regression framework to track multiple overlapping emotional dimensions as continuous trajectories over time.
  arXiv  Detail & Related papers  (2026-06-05T10:40:22Z)
• The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation [66.28943752785906]
  Objective metrics for emotional expressiveness are vital for speech generation, particularly in expressive synthesis and voice conversion.<n>We compute cosine similarity of embeddings from encoders like emotion2vec, assuming they capture affective cues despite linguistic and speaker variations.<n>Despite high classification accuracy, these latent spaces are unsuitable for zero-shot similarity evaluation.
  arXiv  Detail & Related papers  (2026-04-29T06:59:48Z)
• Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition [56.00118641432005]
  We propose a Memory-guided Prototypical Co-occurrence Learning framework that explicitly models emotion co-occurrence patterns.<n>Inspired by human cognitive memory systems, we introduce a memory retrieval strategy to extract semantic-level co-occurrence associations.<n>Our model learns affectively informative representations for accurate emotion distribution prediction.
  arXiv  Detail & Related papers  (2026-02-24T04:11:25Z)
• PersonaDrift: A Benchmark for Temporal Anomaly Detection in Language-Based Dementia Monitoring [0.9668407688201359]
  PersonaDrift is a benchmark designed to evaluate machine learning and statistical methods for detecting progressive changes in daily communication.<n>The benchmark focuses on two forms of longitudinal change that caregivers highlighted as particularly salient: flattened sentiment and off-topic replies.<n>Preliminary results show that flattened sentiment can often be detected with simple statistical models in users with low baseline variability.
  arXiv  Detail & Related papers  (2025-11-20T15:15:00Z)
• Emotions as Ambiguity-aware Ordinal Representations [3.527702696095888]
  We introduce ambiguity-aware ordinal emotion representations, a novel framework that captures both the ambiguity present in emotion annotation and the inherent temporal dynamics of emotional traces.<n>Our results demonstrate that ordinal representations outperform conventional ambiguity-aware models on unbounded labels.
  arXiv  Detail & Related papers  (2025-08-26T16:55:11Z)
• VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection [50.57849622045192]
  We propose VAEmo, an efficient framework for emotion-centric joint VA representation learning with external knowledge injection.<n>VAEmo achieves state-of-the-art performance with a compact design, highlighting the benefit of unified cross-modal encoding and emotion-aware semantic guidance.
  arXiv  Detail & Related papers  (2025-05-05T03:00:51Z)
• Spatiotemporal Emotional Synchrony in Dyadic Interactions: The Role of Speech Conditions in Facial and Vocal Affective Alignment [0.0]
  Non-overlapping speech promotes clearer emotional coordination.<n>Overlapping speech disrupts synchrony.<n>Directionality patterns showed that facial expressions more often preceded speech during turn-taking.
  arXiv  Detail & Related papers  (2025-04-29T14:41:55Z)
• Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation [63.94836524433559]
  DICE-Talk is a framework for disentangling identity with emotion and cooperating emotions with similar characteristics. We develop a disentangled emotion embedder that jointly models audio-visual emotional cues through cross-modal attention. Second, we introduce a correlation-enhanced emotion conditioning module with learnable Emotion Banks. Third, we design an emotion discrimination objective that enforces affective consistency during the diffusion process.
  arXiv  Detail & Related papers  (2025-04-25T05:28:21Z)
• Modelling Emotions in Face-to-Face Setting: The Interplay of Eye-Tracking, Personality, and Temporal Dynamics [1.4645774851707578]
  In this study, we showcase how integrating eye-tracking data, temporal dynamics, and personality traits can substantially enhance the detection of both perceived and felt emotions. Our findings inform the design of future affective computing and human-agent systems.
  arXiv  Detail & Related papers  (2025-03-18T13:15:32Z)
• Dual-Constrained Dynamical Neural ODEs for Ambiguity-aware Continuous Emotion Prediction [15.501509284292476]
  We propose an ambiguity-aware dual-constrained Neural ODE approach to model the dynamics of emotion distributions. In our approach, we utilize ODEs parameterised by neural networks to estimate the distribution parameters, and we integrate additional constraints to restrict the range of the system outputs.
  arXiv  Detail & Related papers  (2024-07-31T05:18:06Z)
• Emotion Recognition from the perspective of Activity Recognition [0.0]
  Appraising human emotional states, behaviors, and reactions displayed in real-world settings can be accomplished using latent continuous dimensions. For emotion recognition systems to be deployed and integrated into real-world mobile and computing devices, we need to consider data collected in the world. We propose a novel three-stream end-to-end deep learning regression pipeline with an attention mechanism.
  arXiv  Detail & Related papers  (2024-03-24T18:53:57Z)
• Dynamic Causal Disentanglement Model for Dialogue Emotion Detection [77.96255121683011]
  We propose a Dynamic Causal Disentanglement Model based on hidden variable separation. This model effectively decomposes the content of dialogues and investigates the temporal accumulation of emotions. Specifically, we propose a dynamic temporal disentanglement model to infer the propagation of utterances and hidden variables.
  arXiv  Detail & Related papers  (2023-09-13T12:58:09Z)
• Unifying the Discrete and Continuous Emotion labels for Speech Emotion Recognition [28.881092401807894]
  In paralinguistic analysis for emotion detection from speech, emotions have been identified with discrete or dimensional (continuous-valued) labels. We propose a model to jointly predict continuous and discrete emotional attributes.
  arXiv  Detail & Related papers  (2022-10-29T16:12:31Z)
• Dilated Context Integrated Network with Cross-Modal Consensus for Temporal Emotion Localization in Videos [128.70585652795637]
  TEL presents three unique challenges compared to temporal action localization. The emotions have extremely varied temporal dynamics. The fine-grained temporal annotations are complicated and labor-intensive.
  arXiv  Detail & Related papers  (2022-08-03T10:00:49Z)
• EEGminer: Discovering Interpretable Features of Brain Activity with Learnable Filters [72.19032452642728]
  We propose a novel differentiable EEG decoding pipeline consisting of learnable filters and a pre-determined feature extraction module. We demonstrate the utility of our model towards emotion recognition from EEG signals on the SEED dataset and on a new EEG dataset of unprecedented size. The discovered features align with previous neuroscience studies and offer new insights, such as marked differences in the functional connectivity profile between left and right temporal areas during music listening.
  arXiv  Detail & Related papers  (2021-10-19T14:22:04Z)
• Few-shot Learning in Emotion Recognition of Spontaneous Speech Using a Siamese Neural Network with Adaptive Sample Pair Formation [11.592365534228895]
  This paper proposes a few-shot learning approach for automatically recognizing emotion in spontaneous speech from a small number of labelled samples. Few-shot learning is implemented via a metric learning approach through a siamese neural network. Results indicate the feasibility of the proposed metric learning in recognizing emotions from spontaneous speech in four datasets.
  arXiv  Detail & Related papers  (2021-09-07T08:04:02Z)
• Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks [82.54695985117783]
  We investigate the suitability of state-of-the-art deep learning architectures for continuous emotion recognition using long video sequences captured in-the-wild. We have developed and evaluated convolutional recurrent neural networks combining 2D-CNNs and long short term-memory units, and inflated 3D-CNN models, which are built by inflating the weights of a pre-trained 2D-CNN model during fine-tuning.
  arXiv  Detail & Related papers  (2020-11-18T13:42:05Z)
• Facial Emotion Recognition with Noisy Multi-task Annotations [88.42023952684052]
  We introduce a new problem of facial emotion recognition with noisy multi-task annotations. For this new problem, we suggest a formulation from the point of joint distribution match view. We exploit a new method to enable the emotion prediction and the joint distribution learning.
  arXiv  Detail & Related papers  (2020-10-19T20:39:37Z)
• Time-Frequency Scattering Accurately Models Auditory Similarities Between Instrumental Playing Techniques [5.923588533979649]
  We show that timbre perception operates within a more flexible taxonomy than those provided by instruments or playing techniques alone. We propose a machine listening model to recover the cluster graph of auditory similarities across instruments, mutes, and techniques.
  arXiv  Detail & Related papers  (2020-07-21T16:37:15Z)
• Facial Expression Editing with Continuous Emotion Labels [76.36392210528105]
  Deep generative models have achieved impressive results in the field of automated facial expression editing. We propose a model that can be used to manipulate facial expressions in facial images according to continuous two-dimensional emotion labels.
  arXiv  Detail & Related papers  (2020-06-22T13:03:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.