How Annotation Trains Annotators: Competence Development in Social Influence Recognition

• URL: http://arxiv.org/abs/2604.02951v1
• Date: Fri, 03 Apr 2026 10:32:57 GMT
• Title: How Annotation Trains Annotators: Competence Development in Social Influence Recognition
• Abstract summary: This study investigates changes in the quality of annotators' work from a competence perspective during a process of social influence recognition.<n>The study involved 25 annotators from five different groups, including both experts and non-experts, who annotated a dataset of 1,021 dialogues.<n>The results indicate a significant increase in annotators' self-perceived competence and confidence.
• Score: 58.27352499161234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human data annotation, especially when involving experts, is often treated as an objective reference. However, many annotation tasks are inherently subjective, and annotators' judgments may evolve over time. This study investigates changes in the quality of annotators' work from a competence perspective during a process of social influence recognition. The study involved 25 annotators from five different groups, including both experts and non-experts, who annotated a dataset of 1,021 dialogues with 20 social influence techniques, along with intentions, reactions, and consequences. An initial subset of 150 texts was annotated twice - before and after the main annotation process - to enable comparison. To measure competence shifts, we combined qualitative and quantitative analyses of the annotated data, semi-structured interviews with annotators, self-assessment surveys, and Large Language Model training and evaluation on the comparison dataset. The results indicate a significant increase in annotators' self-perceived competence and confidence. Moreover, observed changes in data quality suggest that the annotation process may enhance annotator competence and that this effect is more pronounced in expert groups. The observed shifts in annotator competence have a visible impact on the performance of LLMs trained on their annotated data.

Related papers

• Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework [51.56484100374058]
  This work proposes a quantitative approach to narrative assessment by focusing on the linguistic dimension as a primary indicator of quality.<n>The paper presents a methodology for the automatic evaluation of narrative based on the extraction of a comprehensive set of 33 quantitative linguistic features.
  arXiv  Detail & Related papers  (2026-04-21T09:21:40Z)
• ARTICLE: Annotator Reliability Through In-Context Learning [18.818071256242327]
  We propose texttARTICLE, an in-context learning framework to estimate annotation quality through self-consistency. Our findings indicate that texttARTICLE can be used as a robust method for identifying reliable annotators, hence improving data quality.
  arXiv  Detail & Related papers  (2024-09-18T17:59:32Z)
• Annotator in the Loop: A Case Study of In-Depth Rater Engagement to Create a Bridging Benchmark Dataset [1.825224193230824]
  We describe a novel, collaborative, and iterative annotator-in-the-loop methodology for annotation. Our findings indicate that collaborative engagement with annotators can enhance annotation methods.
  arXiv  Detail & Related papers  (2024-08-01T19:11:08Z)
• Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models [52.368110271614285]
  We introduce AdvEval, a novel black-box adversarial framework against NLG evaluators. AdvEval is specially tailored to generate data that yield strong disagreements between human and victim evaluators. We conduct experiments on 12 victim evaluators and 11 NLG datasets, spanning tasks including dialogue, summarization, and question evaluation.
  arXiv  Detail & Related papers  (2024-05-23T14:48:15Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• AQuA -- Combining Experts' and Non-Experts' Views To Assess Deliberation Quality in Online Discussions Using LLMs [0.9737366359397255]
  AQuA is an additive score that calculates a unified deliberative quality score from multiple indices for each discussion post. We develop adapter models for 20 deliberative indices, and calculate correlation coefficients between experts' annotations and the perceived deliberativeness by non-experts to weigh the individual indices into a single deliberative score.
  arXiv  Detail & Related papers  (2024-04-03T14:07:02Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion [7.028604573959653]
  We introduce an innovative annotated dataset for modeling subtle opinion fluctuations and detecting fine-grained stances. The dataset includes a sufficient amount of stance polarity and intensity labels per user over time and within entire conversational threads. All posts are annotated by non-experts and a significant portion of the data is also annotated by experts.
  arXiv  Detail & Related papers  (2022-04-16T09:31:25Z)
• Re-Examining Human Annotations for Interpretable NLP [80.81532239566992]
  We conduct controlled experiments using crowd-sourced websites on two widely used datasets in Interpretable NLP. We compare the annotation results obtained from recruiting workers satisfying different levels of qualification. Our results reveal that the annotation quality is highly subject to the workers' qualification, and workers can be guided to provide certain annotations by the instructions.
  arXiv  Detail & Related papers  (2022-04-10T02:27:30Z)
• Estimating Presentation Competence using Multimodal Nonverbal Behavioral Cues [7.340483819263093]
  Public speaking and presentation competence plays an essential role in many areas of social interaction. One approach that can promote efficient development of presentation competence is the automated analysis of human behavior during a speech. In this work, we investigate the contribution of different nonverbal behavioral cues, namely, facial, body pose-based, and audio-related features, to estimate presentation competence.
  arXiv  Detail & Related papers  (2021-05-06T13:09:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.