VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs

• URL: http://arxiv.org/abs/2603.08936v1
• Date: Mon, 09 Mar 2026 21:10:34 GMT
• Title: VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs
• Abstract summary: Speech Large Language Models (LLMs) show great promise for speech emotion recognition via generative interfaces.<n> shifting from closed-set to open text generation introduces zero-shotity, making evaluation highly sensitive to prompts.<n>We present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs.
• Score: 54.75016325571445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Speech Large Language Models (LLMs) show great promise for speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to prompts. Additionally, conventional speech LLMs benchmarks overlook the inherent ambiguity of human emotion. Hence, we present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs. VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions.

Related papers

• SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis [45.09009423124683]
  Speech Sentiment Analysis plays a critical role in decoding paralinguistic cues for diverse real-world applications.<n>Existing approaches rely on text-centric pipelines that cascade Automatic Speech Recognition with text analysis.<n>We propose a novel dataset, SpeechSense, for fine-grained speech sentiment analysis.
  arXiv  Detail & Related papers  (2026-08-18T15:51:27Z)
• Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation [17.398708962032686]
  Speech-aware large language models (LLMs) can accept speech inputs, yet their training objectives largely emphasize linguistic content or specific fields such as emotions or the speaker's gender.<n>We propose a model-agnostic scoring protocol that produces continuous verification scores for both API-only and open-weight models.
  arXiv  Detail & Related papers  (2026-03-11T14:34:25Z)
• Seeing What You Say: Expressive Image Generation from Speech [39.6782945295833]
  VoxStudio generates expressive images directly from spoken descriptions by jointly aligning linguistic and paralinguistic information.<n>By operating directly on semantic tokens, VoxStudio eliminates the need for an additional speech-to-text system.<n>We also release VoxEmoset, a large-scale paired emotional speech-image dataset built via an advanced TTS engine.
  arXiv  Detail & Related papers  (2025-11-05T12:40:28Z)
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance [66.74042564585942]
  MOSS-Speech is a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance.<n>Our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
  arXiv  Detail & Related papers  (2025-10-01T04:32:37Z)
• Plug-and-Play Emotion Graphs for Compositional Prompting in Zero-Shot Speech Emotion Recognition [3.1649536621597973]
  Large audio-language models (LALMs) exhibit strong zero-shot performance across speech tasks but struggle with speech emotion recognition (SER)<n>We propose Compositional Chain-of-Thought Prompting for Emotion Reasoning (CCoT-Emo) to guide LALMs in emotion inference without fine-tuning.
  arXiv  Detail & Related papers  (2025-09-29T20:06:03Z)
• EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition [0.0]
  Emotion recognition from speech is a challenging task that requires capturing both linguistic and paralinguistic cues.<n>Recent works have highlighted the ability of Large Language Models (LLMs) to perform tasks outside of the sole natural language area.<n>This work proposes a novel approach that fine-tunes an LLM with audio and text representations for emotion prediction.
  arXiv  Detail & Related papers  (2025-08-19T06:58:16Z)
• ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models [70.56468982313834]
  We propose ProsodyLM, which introduces a simple tokenization scheme amenable to learning prosody.<n>We find that ProsodyLM can learn surprisingly diverse emerging prosody processing capabilities through pre-training alone.
  arXiv  Detail & Related papers  (2025-07-27T00:59:01Z)
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study [58.55905182336196]
  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation.<n>We investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling.<n>We introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens.
  arXiv  Detail & Related papers  (2025-06-14T15:26:31Z)
• Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech [29.847183061204436]
  This work studies the capabilities of a large language model (LLM) to understand paralinguistic aspects of speech without fine-tuning its weights.<n>We utilize an end-to-end system with a speech encoder, which is trained to produce token embeddings such that the LLM's response to an expressive speech prompt is aligned with its response to a semantically matching text prompt.
  arXiv  Detail & Related papers  (2024-10-02T01:32:47Z)
• SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts [108.04306136086807]
  We present research that explores the application of prompt tuning to stimulate speech LMs for various generation tasks, within a unified framework called SpeechGen. The proposed unified framework holds great promise for efficiency and effectiveness, particularly with the imminent arrival of advanced speech LMs.
  arXiv  Detail & Related papers  (2023-06-03T22:35:27Z)
• Introducing Semantics into Speech Encoders [91.37001512418111]
  We propose an unsupervised way of incorporating semantic information from large language models into self-supervised speech encoders without labeled audio transcriptions. Our approach achieves similar performance as supervised methods trained on over 100 hours of labeled audio transcripts.
  arXiv  Detail & Related papers  (2022-11-15T18:44:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.