SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis

• URL: http://arxiv.org/abs/2608.17931v1
• Date: Tue, 18 Aug 2026 15:51:27 GMT
• Title: SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis
• Abstract summary: Speech Sentiment Analysis plays a critical role in decoding paralinguistic cues for diverse real-world applications.<n>Existing approaches rely on text-centric pipelines that cascade Automatic Speech Recognition with text analysis.<n>We propose a novel dataset, SpeechSense, for fine-grained speech sentiment analysis.
• Score: 45.09009423124683
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in AI have revolutionized speech processing, yet effective speech understanding requires discerning not just what is said, but how it is said. Speech Sentiment Analysis plays a critical role in decoding these paralinguistic cues for diverse real-world applications such as recruitment and customer service. However, existing Speech Sentiment Analysis research faces two primary limitations. First, dominant approaches rely on text-centric pipelines that cascade Automatic Speech Recognition with text analysis. This process inevitably discards essential acoustic features like prosody and tone, failing to capture attitudinal meanings in acoustically ambiguous utterances. Second, current benchmarks suffer from a mismatch in label granularity, prioritizing basic emotions (e.g., happy, sad) over the nuanced interpersonal stances (e.g., confident, impatient) necessary for social sensitivity. To address these limitations, we propose a novel dataset, SpeechSense, for fine-grained speech sentiment analysis. Specifically, we define a specialized 8-class taxonomy of interpersonal stances detectable primarily through prosodic cues beyond lexical content alone. We then construct a curated dataset based on this taxonomy, built from high-fidelity speech synthesis and rigorous human validation. Comprehensive experiments across multi-modal LLMs, text-only LLMs, and speech encoders demonstrate that models with acoustic access consistently outperform text-only baselines. These results empirically validate the primacy of acoustic cues in detecting subtle speaker attitudes, highlighting the necessity of SpeechSense. Dataset and supplementary materials are available at https://github.com/Sher13cked/SpeechSense.

Related papers

• Speech Codec Probing from Semantic and Phonetic Perspectives [49.01048570474675]
  Speech tokenizers are essential for connecting speech to large language models (LLMs) in multimodal systems.<n> emerging evidence suggests that what is termed "semantic" in speech representations does not align with text-derived semantics.
  arXiv  Detail & Related papers  (2026-03-11T03:32:25Z)
• Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialogue [12.181747090385612]
  Mental manipulation is the strategic use of language to covertly influence or exploit others.<n>We present the first study of mental manipulation detection in spoken dialogues.<n>Using few-shot large audio-language models and human annotation, we evaluate how modality affects detection accuracy and perception.
  arXiv  Detail & Related papers  (2026-01-13T09:02:08Z)
• Benchmarking Contextual and Paralinguistic Reasoning in Speech-LLMs: A Case Study with In-the-Wild Data [46.12417789276609]
  Speech-LLMs have shown impressive performance in tasks like transcription and translation, yet they remain limited in understanding the paralinguistic aspects of speech crucial for social and emotional intelligence.<n>We propose CP-Bench, a benchmark for evaluating speech-LLMs on contextual paralinguistic reasoning.
  arXiv  Detail & Related papers  (2025-09-20T09:26:40Z)
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study [58.55905182336196]
  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation.<n>We investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling.<n>We introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens.
  arXiv  Detail & Related papers  (2025-06-14T15:26:31Z)
• Improving Speech-based Emotion Recognition with Contextual Utterance Analysis and LLMs [2.8728982844941178]
  Speech Emotion Recognition (SER) focuses on identifying emotional states from spoken language. We propose a novel approach that first refines all available transcriptions to ensure data reliability. We then segment each complete conversation into smaller dialogues and use these dialogues as context to predict the emotion of the target utterance within the dialogue.
  arXiv  Detail & Related papers  (2024-10-27T04:23:34Z)
• Acoustic and linguistic representations for speech continuous emotion recognition in call center conversations [2.0653090022137697]
  We explore the use of pre-trained speech representations as a form of transfer learning towards AlloSat corpus. Our experiments confirm the large gain in performance obtained with the use of pre-trained features. Surprisingly, we found that the linguistic content is clearly the major contributor for the prediction of satisfaction.
  arXiv  Detail & Related papers  (2023-10-06T10:22:51Z)
• EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis [49.04496602282718]
  We introduce Expresso, a high-quality expressive speech dataset for textless speech synthesis. This dataset includes both read speech and improvised dialogues rendered in 26 spontaneous expressive styles. We evaluate resynthesis quality with automatic metrics for different self-supervised discrete encoders.
  arXiv  Detail & Related papers  (2023-08-10T17:41:19Z)
• token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text [65.04385919645395]
  token2vec is a novel joint pre-training framework for unpaired speech and text based on discrete representations of speech. Experiments show that token2vec is significantly superior to various speech-only pre-training baselines, with up to 17.7% relative WER reduction.
  arXiv  Detail & Related papers  (2022-10-30T06:38:19Z)
• SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data [100.46303484627045]
  We propose a cross-modal Speech and Language Model (SpeechLM) to align speech and text pre-training with a pre-defined unified representation. Specifically, we introduce two alternative discrete tokenizers to bridge the speech and text modalities. We evaluate SpeechLM on various spoken language processing tasks including speech recognition, speech translation, and universal representation evaluation framework SUPERB.
  arXiv  Detail & Related papers  (2022-09-30T09:12:10Z)
• Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis [67.73554826428762]
  We propose a novel audio-visual speech enhancement framework for high-fidelity telecommunications in AR/VR. Our approach leverages audio-visual speech cues to generate the codes of a neural speech, enabling efficient synthesis of clean, realistic speech from noisy signals.
  arXiv  Detail & Related papers  (2022-03-31T17:57:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.