LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps

• URL: http://arxiv.org/abs/2604.27345v2
• Date: Fri, 01 May 2026 01:59:42 GMT
• Title: LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps
• Abstract summary: Human annotators frequently disagree on emotion labels.<n>Most evaluations of Large Language Model (LLM) emotion annotation collapse these judgments into a single gold standard.<n>We propose three lightweight post-hoc calibration methods that reduce the distributional gap by up to 14%.
• Score: 10.159901538172575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human annotators frequently disagree on emotion labels, yet most evaluations of Large Language Model (LLM) emotion annotation collapse these judgments into a single gold standard, discarding the distributional information that disagreement encodes. We ask whether LLMs capture the structure of this disagreement, not just majority labels, by comparing emotion judgment distributions between human annotators and four zero-shot LLMs, plus a fine-tuned RoBERTa baseline, across two complementary benchmarks: GoEmotions and EmoBank, totaling 640,000 LLM responses. Zero-shot models diverge substantially from human distributions, and in-domain fine-tuning, not model scale, is required to close the gap. We formalize a lexical-grounding gradient through a quantitative transparency score that predicts per-category human--LLM agreement: LLMs reliably capture emotions with explicit lexical markers but systematically fail on pragmatically complex emotions requiring contextual inference, a pattern that replicates across both categorical and continuous emotion frameworks. We further propose three lightweight post-hoc calibration methods that reduce the distributional gap by up to 14\%, and provide actionable guidelines for when LLM emotion annotations can, and cannot, substitute for human labeling.

Related papers

• Exposing Weaknesses in Emotion Recognition in Conversations [3.540702214700106]
  We study Emotion Recognition in Conversations (ERC) using Large Language Models (LLMs) in a zero-shot setting.<n>We show that aggregate metrics mask systematic failures.<n>We introduce an LLM-as-Judge framework that evaluates each emotion independently according to its plausibility in the conversational context.
  arXiv  Detail & Related papers  (2026-09-05T01:40:01Z)
• MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models [6.924490096929709]
  We introduce a new benchmark dataset for visual emotion analysis.<n>We hire $20$ annotators per image and ask them to select all emotions they feel from an image.<n>The resulting dataset contains $10,344$ images with $236,998$ valid votes across eight emotions.
  arXiv  Detail & Related papers  (2026-05-14T09:49:32Z)
• Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking [0.03219669032304847]
  Human Label Variation (HLV) remains underexplored in benchmarks despite rapid progress in large language model development.<n>We introduce an evaluation protocol for multimodal large language model benchmarking that explicitly accounts for two conditions: (1) human label agreement and (2) disagreement.
  arXiv  Detail & Related papers  (2026-03-20T08:29:05Z)
• VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs [54.75016325571445]
  Speech Large Language Models (LLMs) show great promise for speech emotion recognition via generative interfaces.<n> shifting from closed-set to open text generation introduces zero-shotity, making evaluation highly sensitive to prompts.<n>We present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs.
  arXiv  Detail & Related papers  (2026-03-09T21:10:34Z)
• Are Multimodal Large Language Models Good Annotators for Image Tagging? [62.01475514488922]
  This paper aims to analyze the gap between MLLM-generated and human annotations.<n>We propose TagLLM, a novel framework for image tagging, which aims to narrow the gap between MLLM-generated and human annotations.
  arXiv  Detail & Related papers  (2026-02-24T14:53:16Z)
• ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoning [67.22219034602514]
  We introduce ADEPT (Agentic Decoding of Emotion via Evidence Probing Tools), a framework that reframes emotion recognition as a multi-turn inquiry process.<n> ADEPT transforms an SLLM into an agent that maintains an evolving candidate emotion set and adaptively invokes dedicated semantic and acoustic probing tools.<n>We show that ADEPT improves primary emotion accuracy in most settings while substantially improving minor emotion characterization.
  arXiv  Detail & Related papers  (2026-02-13T08:33:37Z)
• Scaling Ambiguity: Augmenting Human Annotation in Speech Emotion Recognition with Audio-Language Models [14.458242760193203]
  Speech Emotion Recognition models typically use single categorical labels, overlooking the inherent ambiguity of human emotions.<n>This paper explores whether Large Audio-Language Models (ALMs) can mitigate the annotation bottleneck by generating high-quality synthetic annotations.<n>We introduce a framework leveraging ALMs to create Synthetic Perceptual Proxies, augmenting human annotations to improve ground-truth distribution reliability.
  arXiv  Detail & Related papers  (2026-01-21T03:32:24Z)
• Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detection [5.731621080995591]
  Hate speech spreads widely online, harming individuals and communities, making automatic detection essential for large-scale moderation.<n>Part of the challenge lies in subjectivity: what one person flags as hate speech, another may see as benign.<n>Large Language Models (LLMs) promise scalable annotation, but prior studies demonstrate that they cannot fully replace human judgement.
  arXiv  Detail & Related papers  (2025-12-10T14:00:48Z)
• Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier [53.55996102181836]
  We propose the Emotional Rationale Verifier (ERV) and an Explanation Reward.<n>Our method guides the model to produce reasoning that is explicitly consistent with the target emotion.<n>We show that our approach not only enhances alignment between explanation and prediction but also empowers MLLMs to deliver emotionally coherent, trustworthy interactions.
  arXiv  Detail & Related papers  (2025-10-27T16:40:17Z)
• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models [108.61337743051483]
  We present MME-Emotion, a systematic benchmark that assesses both emotional understanding and reasoning capabilities of MLLMs.<n>MME-Emotion contains over 6,000 curated video clips with task-specific questioning-answering (QA) pairs, spanning broad scenarios to formulate eight emotional tasks.<n>It incorporates a holistic evaluation suite with hybrid metrics for emotion recognition and reasoning, analyzed through a multi-agent system framework.
  arXiv  Detail & Related papers  (2025-08-11T03:14:55Z)
• Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? [84.32752330104775]
  Variation in human annotation (i.e., disagreements) is common in NLP.<n>We evaluate the influence of different reasoning settings on Large Language Model disagreement modeling.<n>Surprisingly, our results show that RLVR-style reasoning degrades performance in disagreement modeling.
  arXiv  Detail & Related papers  (2025-06-24T09:49:26Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.