Affect Decoding in Phonated and Silent Speech Production from Surface EMG

• URL: http://arxiv.org/abs/2603.11715v2
• Date: Thu, 19 Mar 2026 17:12:02 GMT
• Title: Affect Decoding in Phonated and Silent Speech Production from Surface EMG
• Abstract summary: Measures of articulatory muscle activity such as EMG could reveal how speech production is modulated by emotion alongside acoustic speech analyses.<n>We introduce a dataset comprising 2,780 utterances from 12 participants across 3 tasks, on which we evaluate both intra- and inter-subject decoding.<n>Our results reveal that EMG representations reliably discriminate frustration with up to 0.845 AUC, and generalize well across articulation modes.
• Score: 87.23336551848297
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The expression of affect is integral to spoken communication, yet, its link to underlying articulatory execution remains unclear. Measures of articulatory muscle activity such as EMG could reveal how speech production is modulated by emotion alongside acoustic speech analyses. We investigate affect decoding from facial and neck surface electromyography (sEMG) during phonated and silent speech production. For this purpose, we introduce a dataset comprising 2,780 utterances from 12 participants across 3 tasks, on which we evaluate both intra- and inter-subject decoding using a range of features and model embeddings. Our results reveal that EMG representations reliably discriminate frustration with up to 0.845 AUC, and generalize well across articulation modes. Our ablation study further demonstrates that affective signatures are embedded in facial motor activity and persist in the absence of phonation, highlighting the potential of EMG sensing for affect-aware silent speech interfaces.

Related papers

• Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS [61.29502937013759]
  We introduce a flow-matching based text-to-speech model trained with vocal effort and articulation pseudo-labels.<n>The proposed model achieves continuous and disentangled control of vocal effort and articulation, while also enabling word-level emphasis for clarifying specific segments of an utterance.
  arXiv  Detail & Related papers  (2026-06-22T11:14:22Z)
• CAT-Net: A Cross-Attention Tone Network for Cross-Subject EEG-EMG Fusion Tone Decoding [0.8714814768600078]
  We propose a novel cross-subject multimodal BCI decoding framework.<n>It fuses EEG and EMG signals to classify four Mandarin tones under both audible and silent speech conditions.<n>Our findings suggest that tone-level decoding with minimal EEG-EMG channels is feasible and potentially generalizable across subjects.
  arXiv  Detail & Related papers  (2025-11-14T03:50:54Z)
• A Silent Speech Decoding System from EEG and EMG with Heterogenous Electrode Configurations [0.20075899678041528]
  We introduce neural networks that can handle EEG/EMG with heterogeneous electrode placements.<n>We show strong performance in silent speech decoding via multi-task training on large-scale EEG/EMG datasets.
  arXiv  Detail & Related papers  (2025-06-16T07:57:35Z)
• Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025 [64.59170359368699]
  We present a robust system for the INTERSPEECH 2025 Speech Emotion Recognition in Naturalistic Conditions Challenge.<n>Our method combines state-of-the-art audio models with text features enriched by prosodic and spectral cues.
  arXiv  Detail & Related papers  (2025-06-02T13:46:02Z)
• VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection [50.57849622045192]
  We propose VAEmo, an efficient framework for emotion-centric joint VA representation learning with external knowledge injection.<n>VAEmo achieves state-of-the-art performance with a compact design, highlighting the benefit of unified cross-modal encoding and emotion-aware semantic guidance.
  arXiv  Detail & Related papers  (2025-05-05T03:00:51Z)
• Geometry of orofacial neuromuscular signals: speech articulation decoding using surface electromyography [0.0]
  We present data and methods for decoding speech articulations using surface electromyogram (EMG) signals.<n>EMG-based speech neuroprostheses offer a promising approach for restoring audible speech in individuals who have lost the ability to speak intelligibly.
  arXiv  Detail & Related papers  (2024-11-04T20:31:22Z)
• NeuroSpex: Neuro-Guided Speaker Extraction with Cross-Modal Attention [47.8479647938849]
  We present a neuro-guided speaker extraction model, i.e. NeuroSpex, using the EEG response of the listener as the sole auxiliary reference cue. We propose a novel EEG signal encoder that captures the attention information. Additionally, we propose a cross-attention (CA) mechanism to enhance the speech feature representations.
  arXiv  Detail & Related papers  (2024-09-04T07:33:01Z)
• Exploring Speech Pattern Disorders in Autism using Machine Learning [12.469348589699766]
  This study presents a comprehensive approach to identify distinctive speech patterns through the analysis of examiner-patient dialogues. We extracted 40 speech-related features, categorized into frequency, zero-crossing rate, energy, spectral characteristics, Mel Frequency Cepstral Coefficients (MFCCs) and balance. The classification model aimed to differentiate between ASD and non-ASD cases, achieving an accuracy of 87.75%.
  arXiv  Detail & Related papers  (2024-05-03T02:59:15Z)
• Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues [80.53407593586411]
  We focus on leveraging facial cues beyond the lip region for robust Audio-Visual Speech Enhancement (AVSE) We propose a Dual Attention Cooperative Framework, DualAVSE, to ignore speech-unrelated information, capture speech-related information with facial cues, and dynamically integrate it with the audio signal for AVSE.
  arXiv  Detail & Related papers  (2023-11-24T04:30:31Z)
• Decoding speech perception from non-invasive brain recordings [48.46819575538446]
  We introduce a model trained with contrastive-learning to decode self-supervised representations of perceived speech from non-invasive recordings. Our model can identify, from 3 seconds of MEG signals, the corresponding speech segment with up to 41% accuracy out of more than 1,000 distinct possibilities.
  arXiv  Detail & Related papers  (2022-08-25T10:01:43Z)
• The "Sound of Silence" in EEG -- Cognitive voice activity detection [22.196642357767338]
  "Non-speech"(NS) state of brain activity corresponding to silence regions of speech audio is studied. Speech perception is studied to inspect the existence of such a state, followed by its identification in speech imagination. The recognition performance and the visual distinction observed demonstrates the existence of silence signatures in EEG.
  arXiv  Detail & Related papers  (2020-10-12T07:47:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.