Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

• URL: http://arxiv.org/abs/2606.21069v1
• Date: Fri, 19 Jun 2026 03:36:05 GMT
• Title: Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework
• Abstract summary: We present a multilabel emotion annotation case study.<n>We examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion classifiers.<n>Our results provide practical guidance for designing, aggregating, and evaluating multilabel emotion datasets.
• Score: 0.042970700836450486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Emotion annotation is inherently subjective, yet most NLP pipelines still assume "gold" labels, typically produced by majority voting, and treat annotator variation as noise. In this paper, we present a multilabel emotion annotation case study and use it to examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion classifiers. Rather than collapsing disagreement into a single label, we represent targets as soft vote-share labels (including an intensity-weighted variant) and evaluate models using both thresholded metrics (macro-/micro-F1) and probabilistic alignment (Bernoulli cross-entropy SoftBCE), alongside data-derived disagreement diagnostics. Across annotation regimes, we show that disagreement is structured and leaves measurable traces in model behavior: hard labels may maximize F1 metrics, while soft supervision yields predictions that better reflect empirical annotator variance and uncertainty. Our results provide practical guidance for designing, aggregating, and evaluating multilabel emotion datasets when multiple interpretations are plausible.

Related papers

• Discourse-Role Labels as Presentation-Time Variables for Context Use in Language Models [3.376024929309146]
  We study the effect of Context-augmented language model systems on reader-model behavior.<n>We introduce a paired fixed-content probe over 500 MMLU-Pro items.<n>Each item receives the same misleading answer-bearing assertion under different discourse-role labels.<n>Across GPT-5.5, DeepSeek V4 Pro, Llama-3-8B-Instruct, and Qwen2.5-7B-Instruct, Misleading Adoption Rate shifts by 56-84 percentage points.
  arXiv  Detail & Related papers  (2026-06-02T18:12:57Z)
• Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization [51.93456979139756]
  Free-text explanations extend human label variation beyond label disagreement.<n>We study whether large language models can learn and reproduce such annotator-specific label-explanation behavior.
  arXiv  Detail & Related papers  (2026-05-27T17:55:00Z)
• MLCBART: Multilabel Classification with Bayesian Additive Regression Trees [0.6117371161379209]
  Multilabel Classification deals with the simultaneous classification of multiple binary labels.<n>BART is a nonparametric and flexible model structure capable of uncovering complex relationships within the data.<n>Our adaptation, MLCBART, assumes that labels arise from thresholding an underlying numeric scale.
  arXiv  Detail & Related papers  (2026-01-13T20:17:45Z)
• Noise-Resistant Label Reconstruction Feature Selection for Partial Multi-Label Learning [3.635311806373203]
  "Curse of dimensionality" is prevalent across various data patterns, which increases the risk of model overfitting and leads to a decline in model classification performance.<n>Existing Partial Multi-label Learning (PML) methods addressing this problem are mainly based on the low-rank assumption.<n>In this paper, a PML feature selection method is proposed considering two important characteristics of dataset.
  arXiv  Detail & Related papers  (2025-06-05T06:31:04Z)
• Learning Annotation Consensus for Continuous Emotion Recognition [6.944372188747803]
  In affective computing, datasets often contain multiple annotations from different annotators, which may lack full agreement.<n>We propose a multi-annotator training approach for continuous emotion recognition (CER) that seeks a consensus across all annotators.<n>Our method employs a consensus network to aggregate annotations into a unified representation, guiding the main arousal-valence predictor to better reflect collective inputs.
  arXiv  Detail & Related papers  (2025-05-27T13:43:45Z)
• Label Denoising through Cross-Model Agreement [43.5145547124009]
  Memorizing noisy labels could affect the learning of the model, leading to sub-optimal performances. We propose a novel framework to learn robust machine-learning models from noisy labels.
  arXiv  Detail & Related papers  (2023-08-27T00:31:04Z)
• Exploiting Completeness and Uncertainty of Pseudo Labels for Weakly Supervised Video Anomaly Detection [149.23913018423022]
  Weakly supervised video anomaly detection aims to identify abnormal events in videos using only video-level labels. Two-stage self-training methods have achieved significant improvements by self-generating pseudo labels. We propose an enhancement framework by exploiting completeness and uncertainty properties for effective self-training.
  arXiv  Detail & Related papers  (2022-12-08T05:53:53Z)
• Category-Adaptive Label Discovery and Noise Rejection for Multi-label Image Recognition with Partial Positive Labels [78.88007892742438]
  Training multi-label models with partial positive labels (MLR-PPL) attracts increasing attention. Previous works regard unknown labels as negative and adopt traditional MLR algorithms. We propose to explore semantic correlation among different images to facilitate the MLR-PPL task.
  arXiv  Detail & Related papers  (2022-11-15T02:11:20Z)
• Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priors [24.365876333182207]
  We propose a novel training loss based on per-utterance Dirichlet prior distributions for verbal emotion recognition. An additional metric is used to evaluate the performance by detecting test utterances with high labelling uncertainty. Experiments with the widely used IEMOCAP dataset demonstrate that the two-branch structure achieves state-of-the-art classification results.
  arXiv  Detail & Related papers  (2022-03-08T23:30:01Z)
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning [111.05365744744437]
  Unsupervised contrastive learning labels crops of the same image as positives, and other image crops as negatives. In this work, we first prove that for contrastive learning, inaccurate label assignment heavily impairs its generalization for semantic instance discrimination. Inspired by this theory, we propose a novel self-labeling refinement approach for contrastive learning.
  arXiv  Detail & Related papers  (2021-06-28T14:24:52Z)
• GRACE: Gradient Harmonized and Cascaded Labeling for Aspect-based Sentiment Analysis [90.43089622630258]
  We propose a GRadient hArmonized and CascadEd labeling model (GRACE) to solve these problems. The proposed model achieves consistency improvement on multiple benchmark datasets and generates state-of-the-art results.
  arXiv  Detail & Related papers  (2020-09-22T13:55:34Z)
• Debiased Contrastive Learning [64.98602526764599]
  We develop a debiased contrastive objective that corrects for the sampling of same-label datapoints. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks.
  arXiv  Detail & Related papers  (2020-07-01T04:25:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.