Exposing Weaknesses in Emotion Recognition in Conversations

• URL: http://arxiv.org/abs/2609.05806v1
• Date: Sat, 05 Sep 2026 01:40:01 GMT
• Title: Exposing Weaknesses in Emotion Recognition in Conversations
• Abstract summary: We study Emotion Recognition in Conversations (ERC) using Large Language Models (LLMs) in a zero-shot setting.<n>We show that aggregate metrics mask systematic failures.<n>We introduce an LLM-as-Judge framework that evaluates each emotion independently according to its plausibility in the conversational context.
• Score: 3.540702214700106
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emotion Recognition in Conversations (ERC) aims to identify speakers' emotions in multi-turn dialogue. Accurate emotion recognition can support a wide range of applications, including empathetic conversational agents, mental health support, and educational technologies. While many recent approaches rely on task-specific fine-tuning, such models may exploit dataset-specific cues. A central yet rarely questioned assumption in ERC is that each utterance can be assigned a single unambiguous emotion label. To investigate this assumption, we study ERC using Large Language Models (LLMs) in a zero-shot setting while incorporating preceding conversational turns as context. We show that aggregate metrics mask systematic failures. Errors concentrate around utterances containing negations, exclamations, and interjections. This pattern is consistent across all evaluated models, suggesting limitations in the benchmarks rather than model-specific weaknesses. A controlled re-annotation study involving four human annotators supports this finding: strong agreement is observed in only 35 percent of cases, with neutral utterances dominating high-agreement instances, while many emotional categories fall into low-agreement regimes. These findings suggest that many apparent model errors reflect genuine annotation ambiguity rather than poor emotion understanding. Standard single-label evaluation is therefore insufficient. To address this limitation, we introduce an LLM-as-Judge framework that evaluates each emotion independently according to its plausibility in the conversational context rather than enforcing a single-label decision.

Related papers

• AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence [1.7294100412670375]
  Emotional intelligence (EI) is central to human communication and increasingly important to assess.<n>We introduce AttuneBench, a benchmark grounded in 200 genuine multi-turn human-model conversations.<n>We find that model rankings on emotion recognition, behavioral classification, preference prediction, and judged response quality are largely independent.
  arXiv  Detail & Related papers  (2026-05-20T21:06:05Z)
• VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs [54.75016325571445]
  Speech Large Language Models (LLMs) show great promise for speech emotion recognition via generative interfaces.<n> shifting from closed-set to open text generation introduces zero-shotity, making evaluation highly sensitive to prompts.<n>We present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs.
  arXiv  Detail & Related papers  (2026-03-09T21:10:34Z)
• Emotion Transcription in Conversation: A Benchmark for Capturing Subtle and Complex Emotional States through Natural Language [3.873439892819606]
  We propose a novel task named Emotion Transcription in Conversation (ETC)<n>This task focuses on generating natural language descriptions that accurately reflect speakers' emotional states within conversational contexts.<n>We constructed a Japanese dataset comprising text-based dialogues annotated with participants' self-reported emotional states, described in natural language.
  arXiv  Detail & Related papers  (2026-03-07T10:14:05Z)
• ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoning [67.22219034602514]
  We introduce ADEPT (Agentic Decoding of Emotion via Evidence Probing Tools), a framework that reframes emotion recognition as a multi-turn inquiry process.<n> ADEPT transforms an SLLM into an agent that maintains an evolving candidate emotion set and adaptively invokes dedicated semantic and acoustic probing tools.<n>We show that ADEPT improves primary emotion accuracy in most settings while substantially improving minor emotion characterization.
  arXiv  Detail & Related papers  (2026-02-13T08:33:37Z)
• Decoding Ambiguous Emotions with Test-Time Scaling in Audio-Language Models [18.059483722792077]
  We introduce the first benchmark for ambiguous emotion recognition in speech with ALMs under test-time scaling.<n>Our evaluation systematically compares eight state-of-the-art ALMs and five TTS strategies across three prominent speech emotion datasets.<n>Our benchmark establishes a foundation for developing more robust, context-aware, and emotionally intelligent speech-based AI systems.
  arXiv  Detail & Related papers  (2026-02-01T07:41:57Z)
• SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models [60.72029578488467]
  SpeechR is a unified benchmark for evaluating reasoning over speech in large audio-language models.<n>It evaluates models along three key dimensions: factual retrieval, procedural inference, and normative judgment.<n> Evaluations on eleven state-of-the-art LALMs reveal that high transcription accuracy does not translate into strong reasoning capabilities.
  arXiv  Detail & Related papers  (2025-08-04T03:28:04Z)
• Improving speaker verification robustness with synthetic emotional utterances [14.63248006004598]
  A speaker verification (SV) system offers an authentication service designed to confirm whether a given speech sample originates from a specific speaker.<n>Previous models exhibit high error rates when dealing with emotional utterances compared to neutral ones.<n>This issue primarily stems from the limited availability of labeled emotional speech data.<n>We propose a novel approach employing the CycleGAN framework to serve as a data augmentation method.
  arXiv  Detail & Related papers  (2024-11-30T02:18:26Z)
• Two in One Go: Single-stage Emotion Recognition with Decoupled Subject-context Transformer [78.35816158511523]
  We present a single-stage emotion recognition approach, employing a Decoupled Subject-Context Transformer (DSCT) for simultaneous subject localization and emotion classification. We evaluate our single-stage framework on two widely used context-aware emotion recognition datasets, CAER-S and EMOTIC.
  arXiv  Detail & Related papers  (2024-04-26T07:30:32Z)
• Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems [57.16442740983528]
  Crowdsourced labels play a crucial role in evaluating task-oriented dialogue systems. Previous studies suggest using only a portion of the dialogue context in the annotation process. This study investigates the influence of dialogue context on annotation quality.
  arXiv  Detail & Related papers  (2024-04-15T17:56:39Z)
• Dynamic Causal Disentanglement Model for Dialogue Emotion Detection [77.96255121683011]
  We propose a Dynamic Causal Disentanglement Model based on hidden variable separation. This model effectively decomposes the content of dialogues and investigates the temporal accumulation of emotions. Specifically, we propose a dynamic temporal disentanglement model to infer the propagation of utterances and hidden variables.
  arXiv  Detail & Related papers  (2023-09-13T12:58:09Z)
• Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversation [25.108385802645163]
  We propose a Supervised Prototypical Contrastive Learning (SPCL) loss for the emotion recognition task. We design a difficulty measure function based on the distance between classes and introduce curriculum learning to alleviate the impact of extreme samples. We achieve state-of-the-art results on three widely used benchmarks.
  arXiv  Detail & Related papers  (2022-10-17T03:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.