Quality and Efficiency of Manual Annotation: Pre-annotation Bias

• URL: http://arxiv.org/abs/2306.09307v1
• Date: Thu, 15 Jun 2023 17:41:14 GMT
• Title: Quality and Efficiency of Manual Annotation: Pre-annotation Bias
• Authors: Marie Mikulov\'a, Milan Straka, Jan \v{S}t\v{e}p\'anek, Barbora \v{S}t\v{e}p\'ankov\'a, Jan Haji\v{c}
• Abstract summary: The aim of the experiment is to judge the final annotation quality when pre-annotation is used. The experiment confirmed that the pre-annotation is an efficient tool for faster manual syntactic annotation.
• Score: 1.949293198748152
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents an analysis of annotation using an automatic pre-annotation for a mid-level annotation complexity task -- dependency syntax annotation. It compares the annotation efforts made by annotators using a pre-annotated version (with a high-accuracy parser) and those made by fully manual annotation. The aim of the experiment is to judge the final annotation quality when pre-annotation is used. In addition, it evaluates the effect of automatic linguistically-based (rule-formulated) checks and another annotation on the same data available to the annotators, and their influence on annotation quality and efficiency. The experiment confirmed that the pre-annotation is an efficient tool for faster manual syntactic annotation which increases the consistency of the resulting annotation without reducing its quality.

Related papers

• Feedback-driven object detection and iterative model improvement [2.3700911865675187]
  We present the development and evaluation of a platform designed to interactively improve object detection models. The platform allows uploading and annotating images as well as fine-tuning object detection models. We show evidence for a significant time reduction of up to 53% for semi-automatic compared to manual annotation.
  arXiv  Detail & Related papers  (2024-11-29T16:45:25Z)
• On-the-Fly Point Annotation for Fast Medical Video Labeling [1.890063512530524]
  In medical research, deep learning models rely on high-quality annotated data. The need to adjust two corners makes the process inherently frame-by-frame. We propose an on-the-fly method for live video annotation to enhance the annotation efficiency.
  arXiv  Detail & Related papers  (2024-04-22T16:59:43Z)
• IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models [66.32043210237768]
  This paper introduces an influence-driven selective annotation method. It aims to minimize annotation costs while improving the quality of in-context examples. Experiments confirm the superiority of the proposed method on various benchmarks.
  arXiv  Detail & Related papers  (2023-10-16T22:53:54Z)
• Deep Active Learning with Noisy Oracle in Object Detection [5.5165579223151795]
  We propose a composite active learning framework including a label review module for deep object detection. We show that utilizing part of the annotation budget to correct the noisy annotations partially in the active dataset leads to early improvements in model performance. In our experiments we achieve improvements of up to 4.5 mAP points of object detection performance by incorporating label reviews at equal annotation budget.
  arXiv  Detail & Related papers  (2023-09-30T13:28:35Z)
• Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training [16.740101757982828]
  We propose a pragmatic method that reduces the annotation cost for structured label spaces using active learning. Our approach leverages partial annotation, which reduces labeling costs by selecting only the most informative sub-structures for annotation. We also utilize self-training to incorporate the current model's automatic predictions as pseudo-labels for un-annotated sub-structures.
  arXiv  Detail & Related papers  (2023-05-22T01:58:42Z)
• SciAnnotate: A Tool for Integrating Weak Labeling Sources for Sequence Labeling [55.71459234749639]
  SciAnnotate is a web-based tool for text annotation called SciAnnotate, which stands for scientific annotation tool. Our tool provides users with multiple user-friendly interfaces for creating weak labels. In this study, we take multi-source weak label denoising as an example, we utilized a Bertifying Conditional Hidden Markov Model to denoise the weak label generated by our tool.
  arXiv  Detail & Related papers  (2022-08-07T19:18:13Z)
• Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future [63.99570204416711]
  We reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets. We define a uniform evaluation setup including a new formalization of the annotation error detection task. We release our datasets and implementations in an easy-to-use and open source software package.
  arXiv  Detail & Related papers  (2022-06-05T22:31:45Z)
• Pre-trained Language Models as Re-Annotators [3.193067591317475]
  We investigate how to acquire semantic sensitive annotation representations from Pre-trained Language Models. We fine-tune the Pre-trained Language Models based with cross-validation for annotation correction. We study the re-annotation in relation extraction and create a new manually revised dataset, Re-DocRED.
  arXiv  Detail & Related papers  (2022-05-11T09:28:23Z)
• Re-Examining Human Annotations for Interpretable NLP [80.81532239566992]
  We conduct controlled experiments using crowd-sourced websites on two widely used datasets in Interpretable NLP. We compare the annotation results obtained from recruiting workers satisfying different levels of qualification. Our results reveal that the annotation quality is highly subject to the workers' qualification, and workers can be guided to provide certain annotations by the instructions.
  arXiv  Detail & Related papers  (2022-04-10T02:27:30Z)
• Annotation Curricula to Implicitly Train Non-Expert Annotators [56.67768938052715]
  voluntary studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations. We propose annotation curricula, a novel approach to implicitly train annotators.
  arXiv  Detail & Related papers  (2021-06-04T09:48:28Z)
• Active Learning for Coreference Resolution using Discrete Annotation [76.36423696634584]
  We improve upon pairwise annotation for active learning in coreference resolution. We ask annotators to identify mention antecedents if a presented mention pair is deemed not coreferent. In experiments with existing benchmark coreference datasets, we show that the signal from this additional question leads to significant performance gains per human-annotation hour.
  arXiv  Detail & Related papers  (2020-04-28T17:17:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.