Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization

• URL: http://arxiv.org/abs/2605.28802v1
• Date: Wed, 27 May 2026 17:55:00 GMT
• Title: Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
• Abstract summary: Free-text explanations extend human label variation beyond label disagreement.<n>We study whether large language models can learn and reproduce such annotator-specific label-explanation behavior.
• Score: 51.93456979139756
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Free-text explanations extend human label variation (HLV) beyond label disagreement by revealing the reasoning and preferences behind annotators' decisions. We study whether large language models (LLMs) can learn and reproduce such annotator-specific label-explanation behavior. Using two sentence-pair tasks with four annotators each -- natural language inference and paraphrase judgment -- we first analyze whether annotators exhibit stable individual patterns. We find that such patterns are weak at the single-annotation level due to strong input-content effects, but become detectable after input-content reduction and annotator-level aggregation. We then compare prompting and supervised fine-tuning (SFT) baselines and propose cross-annotator preference optimization (CAPO), which contrasts a target annotator's response with other valid but less target-specific annotations for the same input. Experiments show that prompting is limited and unstable, SFT better captures annotator-specific behavior, and CAPO further improves aggregation-aware imitation and judge-based attribution while preserving target-specific reasoning patterns under human validation. Overall, our results show that HLV can be learned as annotator-specific label-explanation behavior, suggesting a path toward scalable explanation-based annotation grounded in annotator histories rather than labels alone.

Related papers

• Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework [0.042970700836450486]
  We present a multilabel emotion annotation case study.<n>We examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion classifiers.<n>Our results provide practical guidance for designing, aggregating, and evaluating multilabel emotion datasets.
  arXiv  Detail & Related papers  (2026-06-19T03:36:05Z)
• FACTOR: Counterfactual Training-Free Test-Time Adaptation for Open-Vocabulary Object Detection [63.91351553178842]
  FACTOR is a training-free Test-time adaptation for Open-vocabulary object detection.<n>It quantifies attribute sensitivity, semantic relevance, and prediction variation to selectively suppress attribute-dependent predictions.<n>Experiments on PASCAL-C, COCO-C, and FoggyCityscapes show that FACTOR consistently outperforms prior TTA methods.
  arXiv  Detail & Related papers  (2026-05-05T02:31:18Z)
• ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation [2.5819252531158683]
  We introduce textbfReasonScaffold, a scaffolded reasoning annotation protocol that exposes LLM-generated explanations while withholding predicted labels.<n>We study how reasoning affects human annotation behavior in a controlled setting, rather than evaluating annotation accuracy.<n>Our results show that exposure to reasoning is associated with increased agreement, along with minimal revision, suggesting that reasoning helps resolve ambiguous cases without inducing widespread changes.
  arXiv  Detail & Related papers  (2026-03-22T07:14:27Z)
• QuMAB: Query-based Multi-Annotator Behavior Modeling with Reliability under Sparse Labels [23.555446749682467]
  Multi-annotator learning traditionally aggregates diverse annotations to approximate a single ground truth, treating disagreements as noise.<n>We introduce a paradigm shift from sample-wise aggregation to annotator-wise behavior modeling.<n>By treating annotator disagreements as valuable information rather than noise, modeling annotator-specific behavior patterns can reconstruct unlabeled data to reduce annotation cost, enhance aggregation reliability, and explain annotator decision behavior.
  arXiv  Detail & Related papers  (2025-07-23T16:17:43Z)
• Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation [60.18907916989796]
  Large Language Models (LLMs) generate chains of thought (CoTs) before giving the final answer.<n>We propose a novel pipeline enriched with linguistically-grounded discourse segmenters to extract supporting and opposing statements for each answer option.<n>We also propose a rank-based HLV evaluation framework that prioritizes the ranking of answers over exact scores.
  arXiv  Detail & Related papers  (2025-05-29T11:47:18Z)
• Subjective Logic Encodings [20.458601113219697]
  Data perspectivism seeks to leverage inter-annotator disagreement to learn models.<n>Subjective Logic SLEs is a framework for constructing classification targets that explicitly encodes annotations as opinions of the annotators.
  arXiv  Detail & Related papers  (2025-02-17T15:14:10Z)
• From Weak to Strong Sound Event Labels using Adaptive Change-Point Detection and Active Learning [11.312115846980602]
  We propose an adaptive change point detection method (A-CPD) for machine guided weak label annotation of audio recording segments. For each unlabeled audio recording, we use a prediction model to derive a probability curve used to guide annotation. We show that it is possible to derive strong labels of high quality with a limited annotation budget, and show favorable results for A-CPD.
  arXiv  Detail & Related papers  (2024-03-13T13:33:35Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• AnnoBERT: Effectively Representing Multiple Annotators' Label Choices to Improve Hate Speech Detection [18.823219608659986]
  AnnoBERT is a first-of-its-kind architecture integrating annotator characteristics and label text to detect hate speech. During training, the model associates annotators with their label choices given a piece of text. During evaluation, when label information is not available, the model predicts the aggregated label given by the participating annotators.
  arXiv  Detail & Related papers  (2022-12-20T16:30:11Z)
• Exploiting Completeness and Uncertainty of Pseudo Labels for Weakly Supervised Video Anomaly Detection [149.23913018423022]
  Weakly supervised video anomaly detection aims to identify abnormal events in videos using only video-level labels. Two-stage self-training methods have achieved significant improvements by self-generating pseudo labels. We propose an enhancement framework by exploiting completeness and uncertainty properties for effective self-training.
  arXiv  Detail & Related papers  (2022-12-08T05:53:53Z)
• Self-Supervised Tuning for Few-Shot Segmentation [82.32143982269892]
  Few-shot segmentation aims at assigning a category label to each image pixel with few annotated samples. Existing meta-learning method tends to fail in generating category-specifically discriminative descriptor when the visual features extracted from support images are marginalized in embedding space. This paper presents an adaptive framework tuning, in which the distribution of latent features across different episodes is dynamically adjusted based on a self-segmentation scheme.
  arXiv  Detail & Related papers  (2020-04-12T03:53:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.