Looking for Affect in Spontaneous Finnish Speech through Linguistic Interpretability

• URL: http://arxiv.org/abs/2607.24155v1
• Date: Mon, 27 Jul 2026 08:32:40 GMT
• Title: Looking for Affect in Spontaneous Finnish Speech through Linguistic Interpretability
• Abstract summary: We show that the combination of text- and audio-based features improves valence regression results over the individual modalities.<n>Results support prior findings from other languages, providing new data and knowledge on spontaneous Finnish speech.
• Score: 4.94997283141601
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing research on affect in speech has shown how acoustic surface characteristics and content-related linguistic aspects of speech both relate to perceived emotional arousal and valence. However, it is not clear what the relative contributions of these two factors are in the perceptual process. This is especially true for Finnish, for which most existing studies focus on either acoustic-phonetic or text analysis. This paper presents a study where we systematically explore the combinatory role of text- and audio-based features in modeling the human perception of valence and arousal using a newly released affective speech corpus for spontaneous Finnish. We show that the combination of text- and audio-based features improves valence regression results over the individual modalities, whereas for arousal regression the complementary effect is not substantial. The results support prior findings from other languages, providing new data and knowledge on spontaneous Finnish speech.

Related papers

• iMiGUE-Speech: A Spontaneous Speech Dataset for Affective Analysis [7.298729249943839]
  iMiGUE-Speech is an extension of the iMiGUE dataset that provides a spontaneous affective corpus for studying emotional and affective states.<n>iMiGUE-Speech captures spontaneous affect arising naturally from real match outcomes.
  arXiv  Detail & Related papers  (2026-02-25T00:38:19Z)
• EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Models [47.41816926003011]
  Speech Language Models (SLMs) have made significant progress in spoken language understanding.<n>It remains unclear whether SLMs can fully perceive non lexical vocal cues alongside spoken words, and respond with empathy that aligns with both emotional and contextual factors.<n>We present EchoMind, the first interrelated, multi-level benchmark that simulates the cognitive process of empathetic dialogue.
  arXiv  Detail & Related papers  (2025-10-26T17:15:56Z)
• Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio [52.859261069569165]
  We propose the first unified framework capable of handling diverse combinations of sign language, lip movements, and audio for spoken-language text generation.<n>We focus on three main objectives: (i) designing a unified, modality-agnostic architecture capable of effectively processing heterogeneous inputs; (ii) exploring the underexamined synergy among modalities, particularly the role of lip movements as non-manual cues in sign language comprehension; and (iii) achieving performance on par with or better than state-of-the-art models specialized for individual tasks.
  arXiv  Detail & Related papers  (2025-08-28T06:51:42Z)
• On the Relationship between Accent Strength and Articulatory Features [26.865464238029748]
  This paper explores the relationship between accent strength and articulatory features inferred from acoustic speech.<n>The proposed framework leverages recent self-supervised learning articulatory inversion techniques to estimate articulatory features.<n>Results indicate that tongue positioning patterns distinguish the two dialects, with notable differences inter-dialects in rhotic and low back vowels.
  arXiv  Detail & Related papers  (2025-07-03T20:08:28Z)
• Investigating Affect Mining Techniques for Annotation Sample Selection in the Creation of Finnish Affective Speech Corpus [8.008453432425364]
  This paper presents the first such corpus, created by annotating 12,000 utterances for emotional arousal and valence, sampled from three large-scale Finnish speech corpora.<n>As an outcome, the work introduces a spontaneous Finnish affective speech corpus and informs sampling strategies for affective speech corpus creation in other languages or domains.
  arXiv  Detail & Related papers  (2025-05-23T12:47:21Z)
• Acoustic and linguistic representations for speech continuous emotion recognition in call center conversations [2.0653090022137697]
  We explore the use of pre-trained speech representations as a form of transfer learning towards AlloSat corpus. Our experiments confirm the large gain in performance obtained with the use of pre-trained features. Surprisingly, we found that the linguistic content is clearly the major contributor for the prediction of satisfaction.
  arXiv  Detail & Related papers  (2023-10-06T10:22:51Z)
• Measuring the Impact of Individual Domain Factors in Self-Supervised Pre-Training [60.825471653739555]
  We show that phonetic domain factors play an important role during pre-training while grammatical and syntactic factors are far less important. This is the first study to better understand the domain characteristics of pre-trained sets in self-supervised pre-training for speech.
  arXiv  Detail & Related papers  (2022-03-01T17:40:51Z)
• E-ffective: A Visual Analytic System for Exploring the Emotion and Effectiveness of Inspirational Speeches [57.279044079196105]
  E-ffective is a visual analytic system allowing speaking experts and novices to analyze both the role of speech factors and their contribution in effective speeches. Two novel visualizations include E-spiral (that shows the emotional shifts in speeches in a visually compact way) and E-script (that connects speech content with key speech delivery information.
  arXiv  Detail & Related papers  (2021-10-28T06:14:27Z)
• Perception Point: Identifying Critical Learning Periods in Speech for Bilingual Networks [58.24134321728942]
  We compare and identify cognitive aspects on deep neural-based visual lip-reading models. We observe a strong correlation between these theories in cognitive psychology and our unique modeling.
  arXiv  Detail & Related papers  (2021-10-13T05:30:50Z)
• Decomposing lexical and compositional syntax and semantics with deep language models [82.81964713263483]
  The activations of language transformers like GPT2 have been shown to linearly map onto brain activity during speech comprehension. Here, we propose a taxonomy to factorize the high-dimensional activations of language models into four classes: lexical, compositional, syntactic, and semantic representations. The results highlight two findings. First, compositional representations recruit a more widespread cortical network than lexical ones, and encompass the bilateral temporal, parietal and prefrontal cortices.
  arXiv  Detail & Related papers  (2021-03-02T10:24:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.