An Analysis of Active Learning Algorithms using Real-World Crowd-sourced Text Annotations

• URL: http://arxiv.org/abs/2604.23290v1
• Date: Sat, 25 Apr 2026 13:07:07 GMT
• Title: An Analysis of Active Learning Algorithms using Real-World Crowd-sourced Text Annotations
• Abstract summary: Active learning algorithms automatically identify the most informative samples from large amounts of unlabeled data.<n>Existing research on active learning with noisy oracles typically simulate the oracles using machine learning models.<n>In this research, we first collect annotations of text samples from 3 benchmark text classification datasets.
• Score: 19.453809893120184
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Active learning algorithms automatically identify the most informative samples from large amounts of unlabeled data and tremendously reduce human annotation effort in inducing a machine learning model. In a conventional active learning setup, the labeling oracles are assumed to be infallible, that is, they always provide correct answers (in terms of class labels) to the queried unlabeled instances, which cannot be guaranteed in real-world applications. To this end, a body of research has focused on the development of active learning algorithms in the presence of imperfect / noisy oracles. Existing research on active learning with noisy oracles typically simulate the oracles using machine learning models; however, real-world situations are much more challenging, and using ML models to simulate the annotation patterns may not appropriately capture the nuances of real-world annotation challenges. In this research, we first collect annotations of text samples (from 3 benchmark text classification datasets) from crowd-sourced workers through a crowd-sourcing platform. We then conduct extensive empirical studies of 8 commonly used active learning techniques (in conjunction with deep neural networks) using the obtained annotations. Our analyses sheds light on the performance of these techniques under real-world challenges, where annotators can provide incorrect labels, and can also refuse to provide labels. We hope this research will provide valuable insights that will be useful for the deployment of deep active learning systems in real-world applications. The obtained annotations can be accessed at https://github.com/varuntotakura/al_rcta/.

Related papers

• Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems [0.6267574471145215]
  We propose a novel active learning framework with significant potential for application in modern AI systems.<n>Unlike the traditional active learning methods, which only focus on determining which data point should be labeled, our framework also introduces an innovative perspective on incorporating different query scheme.<n>Our proposed active learning framework exhibits higher accuracy and lower loss compared to other methods.
  arXiv  Detail & Related papers  (2024-12-31T05:12:51Z)
• BOWL: A Deceptively Simple Open World Learner [20.856520787551453]
  Traditional machine learning excels on static benchmarks, but the real world is dynamic and seldom as carefully curated as test sets.<n>In this paper, we posit that neural networks already contain a powerful catalyst to turn them into open world learners.<n>We derive effective strategies to detect in- and out-of-distribution samples, select informative data points, and update the model continuously.
  arXiv  Detail & Related papers  (2024-02-07T13:04:35Z)
• Unleashing the Potential of Regularization Strategies in Learning with Noisy Labels [65.92994348757743]
  We demonstrate that a simple baseline using cross-entropy loss, combined with widely used regularization strategies can outperform state-of-the-art methods. Our findings suggest that employing a combination of regularization strategies can be more effective than intricate algorithms in tackling the challenges of learning with noisy labels.
  arXiv  Detail & Related papers  (2023-07-11T05:58:20Z)
• Evaluating Zero-cost Active Learning for Object Detection [4.106771265655055]
  Object detection requires substantial labeling effort for learning robust models. Active learning can reduce this effort by intelligently selecting relevant examples to be annotated. We show that a key ingredient is not only the score on a bounding box level but also the technique used for aggregating the scores for ranking images.
  arXiv  Detail & Related papers  (2022-12-08T11:48:39Z)
• Representation Learning for the Automatic Indexing of Sound Effects Libraries [79.68916470119743]
  We show that a task-specific but dataset-independent representation can successfully address data issues such as class imbalance, inconsistent class labels, and insufficient dataset size. Detailed experimental results show the impact of metric learning approaches and different cross-dataset training methods on representational effectiveness.
  arXiv  Detail & Related papers  (2022-08-18T23:46:13Z)
• A Survey of Learning on Small Data: Generalization, Optimization, and Challenge [101.27154181792567]
  Learning on small data that approximates the generalization ability of big data is one of the ultimate purposes of AI. This survey follows the active sampling theory under a PAC framework to analyze the generalization error and label complexity of learning on small data. Multiple data applications that may benefit from efficient small data representation are surveyed.
  arXiv  Detail & Related papers  (2022-07-29T02:34:19Z)
• ALBench: A Framework for Evaluating Active Learning in Object Detection [102.81795062493536]
  This paper contributes an active learning benchmark framework named as ALBench for evaluating active learning in object detection. Developed on an automatic deep model training system, this ALBench framework is easy-to-use, compatible with different active learning algorithms, and ensures the same training and testing protocols.
  arXiv  Detail & Related papers  (2022-07-27T07:46:23Z)
• What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks? [59.51457877578138]
  We study contrastive learning on the wearable-based activity recognition task. This paper presents an open-source PyTorch library textttCL-HAR, which can serve as a practical tool for researchers.
  arXiv  Detail & Related papers  (2022-02-12T06:10:15Z)
• Low-Regret Active learning [64.36270166907788]
  We develop an online learning algorithm for identifying unlabeled data points that are most informative for training. At the core of our work is an efficient algorithm for sleeping experts that is tailored to achieve low regret on predictable (easy) instances.
  arXiv  Detail & Related papers  (2021-04-06T22:53:45Z)
• A Survey on Deep Learning with Noisy Labels: How to train your model when you cannot trust on the annotations? [21.562089974755125]
  Several approaches have been proposed to improve the training of deep learning models in the presence of noisy labels. This paper presents a survey on the main techniques in literature, in which we classify the algorithm in the following groups: robust losses, sample weighting, sample selection, meta-learning, and combined approaches.
  arXiv  Detail & Related papers  (2020-12-05T15:45:20Z)
• Bayesian active learning for production, a systematic study and a reusable library [85.32971950095742]
  In this paper, we analyse the main drawbacks of current active learning techniques. We do a systematic study on the effects of the most common issues of real-world datasets on the deep active learning process. We derive two techniques that can speed up the active learning loop such as partial uncertainty sampling and larger query size.
  arXiv  Detail & Related papers  (2020-06-17T14:51:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.