Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025

• URL: http://arxiv.org/abs/2606.02255v1
• Date: Mon, 01 Jun 2026 13:43:10 GMT
• Title: Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
• Abstract summary: We provide the first large-scale, task-level audit of human annotation reporting across major NLP venues.<n>We introduce a unified taxonomy of annotation-reporting practices and validate an LLM-assisted extraction pipeline.<n>Our results show that annotation reporting in NLP has improved over time but remains uneven.
• Score: 27.03245242280755
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled. We provide the first large-scale, task-level audit of human annotation reporting across major NLP venues, asking which annotation details are documented, which are missing, and how reporting varies across time, topic, venue, and intended use of human judgment. We introduce a unified taxonomy of annotation-reporting practices and validate an LLM-assisted extraction pipeline against Annotated-gold, a human-adjudicated gold standard of 41 papers and 72 annotation tasks, where the best model reaches human-comparable agreement with adjudicated labels, with Krippendorff's alpha of 0.606 versus 0.585 for human-human agreement. Using this pipeline, we construct Annotated-llm, a dataset covering ACL-venue papers from 2018-2025, with 2,667 extracted annotation tasks from 1,603 papers, and find that papers frequently report operational details such as recruitment strategies, annotator expertise, and annotation volume, but often omit details needed to assess annotation validity, including training, language proficiency, compensation, socio-demographics, adjudication, and agreement values, especially in model-evaluation studies. Our results show that annotation reporting in NLP has improved over time but remains uneven, and they establish a scalable framework and bare-minimum reporting recommendations for making human annotation more reliable, reproducible, and interpretable.

Related papers

• Data Annotation as Measurement [5.058544031889297]
  We argue that data annotation should be understood as a measurement problem.<n>Like other forms of measurement, annotation requires defining a concept, operationalizing it through an instrument, and evaluating the reliability and validity of the resulting measurements.
  arXiv  Detail & Related papers  (2026-08-07T14:51:27Z)
• Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation [16.435716847925313]
  We conduct a large-scale analysis of human evaluation protocols for evaluating long-form generation tasks.<n>We find widespread under-reporting of important aspects of human evaluation study design.
  arXiv  Detail & Related papers  (2026-06-06T01:55:56Z)
• ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation [2.5819252531158683]
  We introduce textbfReasonScaffold, a scaffolded reasoning annotation protocol that exposes LLM-generated explanations while withholding predicted labels.<n>We study how reasoning affects human annotation behavior in a controlled setting, rather than evaluating annotation accuracy.<n>Our results show that exposure to reasoning is associated with increased agreement, along with minimal revision, suggesting that reasoning helps resolve ambiguous cases without inducing widespread changes.
  arXiv  Detail & Related papers  (2026-03-22T07:14:27Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators [8.244694683982784]
  We investigate the questions of assessing the performance of human annotators and incentivizing them to provide high-quality annotations.<n>We develop a principal-agent model to characterize the behaviors of and the interactions between the company and the human annotators.<n>The model rationalizes a practical mechanism of a bonus scheme to incentivize annotators which benefits both parties.
  arXiv  Detail & Related papers  (2025-02-10T12:15:27Z)
• FABLES: Evaluating faithfulness and content selection in book-length summarization [55.50680057160788]
  In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on book-length documents. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate.
  arXiv  Detail & Related papers  (2024-04-01T17:33:38Z)
• FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction [85.26780391682894]
  We propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE) FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation.
  arXiv  Detail & Related papers  (2024-03-04T17:57:18Z)
• LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization [48.02158981582502]
  LongEval is a set of guidelines for human evaluation of faithfulness in long-form summaries. We deploy LongEval in annotation studies on two long-form summarization datasets in different domains.
  arXiv  Detail & Related papers  (2023-01-30T21:31:48Z)
• Active Learning for Abstractive Text Summarization [50.79416783266641]
  We propose the first effective query strategy for Active Learning in abstractive text summarization. We show that using our strategy in AL annotation helps to improve the model performance in terms of ROUGE and consistency scores.
  arXiv  Detail & Related papers  (2023-01-09T10:33:14Z)
• Re-Examining Human Annotations for Interpretable NLP [80.81532239566992]
  We conduct controlled experiments using crowd-sourced websites on two widely used datasets in Interpretable NLP. We compare the annotation results obtained from recruiting workers satisfying different levels of qualification. Our results reveal that the annotation quality is highly subject to the workers' qualification, and workers can be guided to provide certain annotations by the instructions.
  arXiv  Detail & Related papers  (2022-04-10T02:27:30Z)
• Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss [2.9176780332337486]
  We study learning named entity recognizers in the presence of missing entity annotations. We propose a novel loss, the Expected Entity Ratio, to learn models in the presence of systematically missing tags. We show that our approach is both theoretically sound and empirically useful.
  arXiv  Detail & Related papers  (2021-08-16T16:53:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.