Task Decomposition for Efficient Annotation

• URL: http://arxiv.org/abs/2606.24734v1
• Date: Tue, 23 Jun 2026 15:58:46 GMT
• Title: Task Decomposition for Efficient Annotation
• Abstract summary: High-quality annotations of structured representations are expensive to collect over large corpora.<n>In traditional annotation of structure, each complete example is performed end-to-end by a single annotator.<n>We propose to decompose annotation tasks into sub-tasks in order to reduce the aggregate inferential load of annotation projects.
• Score: 10.511472749834581
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-quality annotations of structured representations are expensive to collect over large corpora. Manual annotation of structure is laborious, and model-based annotation, although cheaper to generate, requires expensive validation and potentially significant supervision to ensure that the annotation quality is strong enough to be useful downstream. In traditional annotation workflows, annotation of each complete example is performed end-to-end by a single annotator. However, structured annotation is complex, and each aspect of the task represents a unique challenge with an associated inferential load for a given annotator. Modern annotation projects can incorporate heterogeneous groups of annotators, including both models and human annotators with varying domain and linguistic expertise. It remains unclear, however, how to redesign annotation tasks in this setting, where efforts are discriminately allocated across heterogeneous annotators with respect to distinct annotation challenges. We propose to decompose annotation tasks into sub-tasks in order to reduce the aggregate inferential load of annotation projects. Inspired by the notion of centers from centering theory, we introduce a formal model of inferential load based on the degrees of freedom in the space of valid annotations. Using this model, we show that identifying these centers (i.e. salient anchor entities realized by annotation sub-tasks) constrains the output space complexity, and decompositions which isolate and advance center identification reduce the aggregate inferential load. We provide guidelines for decomposing complex structured annotation tasks, supported by examples demonstrating improved cost-efficiency from our prior work. Finally, we present a procedure for allocating sub-tasks across annotators to maximize quality under a fixed budget.

Related papers

• Weakly-Supervised Cross-Domain Segmentation of Electron Microscopy with Sparse Point Annotation [1.124958340749622]
  We introduce a multitask learning framework to leverage correlations among the counting, detection, and segmentation tasks. We develop a cross-position cut-and-paste for label augmentation and an entropy-based pseudo-label selection. The proposed model is capable of significantly outperforming UDA methods and produces comparable performance as the supervised counterpart.
  arXiv  Detail & Related papers  (2024-03-31T12:22:23Z)
• An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models [55.01592097059969]
  Supervised finetuning on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool. We propose using experimental design to circumvent the computational bottlenecks of active learning.
  arXiv  Detail & Related papers  (2024-01-12T16:56:54Z)
• Deep Active Learning with Noisy Oracle in Object Detection [5.5165579223151795]
  We propose a composite active learning framework including a label review module for deep object detection. We show that utilizing part of the annotation budget to correct the noisy annotations partially in the active dataset leads to early improvements in model performance. In our experiments we achieve improvements of up to 4.5 mAP points of object detection performance by incorporating label reviews at equal annotation budget.
  arXiv  Detail & Related papers  (2023-09-30T13:28:35Z)
• Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training [16.740101757982828]
  We propose a pragmatic method that reduces the annotation cost for structured label spaces using active learning. Our approach leverages partial annotation, which reduces labeling costs by selecting only the most informative sub-structures for annotation. We also utilize self-training to incorporate the current model's automatic predictions as pseudo-labels for un-annotated sub-structures.
  arXiv  Detail & Related papers  (2023-05-22T01:58:42Z)
• Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns [3.1318537187387787]
  We propose a novel approach to determine annotation strategies for segmentation datasets. Our method sequentially determines proportions of segmentation and classification annotations to collect for budget-fractions. We show in our experiments that our approach yields annotations that perform very close to the optimal for a number of different annotation budgets and datasets.
  arXiv  Detail & Related papers  (2023-03-21T08:41:54Z)
• Domain Adaptive Segmentation of Electron Microscopy with Sparse Point Annotations [2.5137859989323537]
  We develop a highly annotation-efficient approach with competitive performance. We focus on weakly-supervised domain adaptation (WDA) with a type of extremely sparse and weak annotation. We show that our model with only 15% point annotations can achieve comparable performance as supervised models.
  arXiv  Detail & Related papers  (2022-10-24T10:50:37Z)
• Dynamic Supervisor for Cross-dataset Object Detection [52.95818230087297]
  Cross-dataset training in object detection tasks is complicated because the inconsistency in the category range across datasets transforms fully supervised learning into semi-supervised learning. We propose a dynamic supervisor framework that updates the annotations multiple times through multiple-updated submodels trained using hard and soft labels. In the final generated annotations, both recall and precision improve significantly through the integration of hard-label training with soft-label training.
  arXiv  Detail & Related papers  (2022-04-01T03:18:46Z)
• Annotation Curricula to Implicitly Train Non-Expert Annotators [56.67768938052715]
  voluntary studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations. We propose annotation curricula, a novel approach to implicitly train annotators.
  arXiv  Detail & Related papers  (2021-06-04T09:48:28Z)
• Semi-Supervised Crowd Counting via Self-Training on Surrogate Tasks [50.78037828213118]
  This paper tackles the semi-supervised crowd counting problem from the perspective of feature learning. We propose a novel semi-supervised crowd counting method which is built upon two innovative components.
  arXiv  Detail & Related papers  (2020-07-07T05:30:53Z)
• Interpretable Entity Representations through Large-Scale Typing [61.4277527871572]
  We present an approach to creating entity representations that are human readable and achieve high performance out of the box. Our representations are vectors whose values correspond to posterior probabilities over fine-grained entity types. We show that it is possible to reduce the size of our type set in a learning-based way for particular domains.
  arXiv  Detail & Related papers  (2020-04-30T23:58:03Z)
• Towards Using Count-level Weak Supervision for Crowd Counting [55.58468947486247]
  This paper studies the problem of weakly-supervised crowd counting which learns a model from only a small amount of location-level annotations (fully-supervised) but a large amount of count-level annotations (weakly-supervised) We devise a simple-yet-effective training strategy, namely Multiple Auxiliary Tasks Training (MATT), to construct regularizes for restricting the freedom of the generated density maps.
  arXiv  Detail & Related papers  (2020-02-29T02:58:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.