Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems

• URL: http://arxiv.org/abs/2501.00277v1
• Date: Tue, 31 Dec 2024 05:12:51 GMT
• Title: Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems
• Authors: Yiran Huang, Jian-Feng Yang, Haoda Fu,
• Abstract summary: We propose a novel active learning framework with significant potential for application in modern AI systems. Unlike the traditional active learning methods, which only focus on determining which data point should be labeled, our framework also introduces an innovative perspective on incorporating different query scheme. Our proposed active learning framework exhibits higher accuracy and lower loss compared to other methods.
• Score: 0.6267574471145215
• License:
• Abstract: Modern AI algorithms require labeled data. In real world, majority of data are unlabeled. Labeling the data are costly. this is particularly true for some areas requiring special skills, such as reading radiology images by physicians. To most efficiently use expert's time for the data labeling, one promising approach is human-in-the-loop active learning algorithm. In this work, we propose a novel active learning framework with significant potential for application in modern AI systems. Unlike the traditional active learning methods, which only focus on determining which data point should be labeled, our framework also introduces an innovative perspective on incorporating different query scheme. We propose a model to integrate the information from different types of queries. Based on this model, our active learning frame can automatically determine how the next question is queried. We further developed a data driven exploration and exploitation framework into our active learning method. This method can be embedded in numerous active learning algorithms. Through simulations on five real-world datasets, including a highly complex real image task, our proposed active learning framework exhibits higher accuracy and lower loss compared to other methods.

Related papers

• Image Classification with Deep Reinforcement Active Learning [28.924413229981827]
  In many real-world scenarios, labeled data are scarce, and their hand-labeling is time, effort and cost demanding. Active learning is an alternative paradigm that mitigates the effort in hand-labeling data, and annotated by an expert. In this work, we devise an adaptive active learning method based on Markov Decision Process (MDP)
  arXiv  Detail & Related papers  (2024-12-27T18:37:51Z)
• Batch Active Learning of Reward Functions from Human Preferences [33.39413552270375]
  Preference-based learning enables reliable labeling by querying users with preference questions. Active querying methods are commonly employed in preference-based learning to generate more informative data. We develop a set of novel algorithms that enable efficient learning of reward functions using as few data samples as possible.
  arXiv  Detail & Related papers  (2024-02-24T08:07:48Z)
• Learning from the Best: Active Learning for Wireless Communications [9.523381807291049]
  Active learning algorithms identify the most critical and informative samples in an unlabeled dataset and label only those samples, instead of the complete set. We present a case study of deep learning-based mmWave beam selection, where labeling is performed by a compute-intensive algorithm based on exhaustive search. Our results show that using an active learning algorithm for class-imbalanced datasets can reduce labeling overhead by up to 50% for this dataset.
  arXiv  Detail & Related papers  (2024-01-23T12:21:57Z)
• Learn to Explore: on Bootstrapping Interactive Data Exploration with Meta-learning [8.92180350317399]
  We propose a learning-to-explore framework, based on meta-learning, which learns how to learn a classifier with automatically generated meta-tasks. Our proposal outperforms existing explore-by-example solutions in terms of accuracy and efficiency.
  arXiv  Detail & Related papers  (2022-12-07T03:12:41Z)
• ALBench: A Framework for Evaluating Active Learning in Object Detection [102.81795062493536]
  This paper contributes an active learning benchmark framework named as ALBench for evaluating active learning in object detection. Developed on an automatic deep model training system, this ALBench framework is easy-to-use, compatible with different active learning algorithms, and ensures the same training and testing protocols.
  arXiv  Detail & Related papers  (2022-07-27T07:46:23Z)
• What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks? [59.51457877578138]
  We study contrastive learning on the wearable-based activity recognition task. This paper presents an open-source PyTorch library textttCL-HAR, which can serve as a practical tool for researchers.
  arXiv  Detail & Related papers  (2022-02-12T06:10:15Z)
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation [64.43440450794495]
  We conduct an extensive study of six offline learning algorithms for robot manipulation. Our study analyzes the most critical challenges when learning from offline human data. We highlight opportunities for learning from human datasets.
  arXiv  Detail & Related papers  (2021-08-06T20:48:30Z)
• Low-Regret Active learning [64.36270166907788]
  We develop an online learning algorithm for identifying unlabeled data points that are most informative for training. At the core of our work is an efficient algorithm for sleeping experts that is tailored to achieve low regret on predictable (easy) instances.
  arXiv  Detail & Related papers  (2021-04-06T22:53:45Z)
• Diverse Complexity Measures for Dataset Curation in Self-driving [80.55417232642124]
  We propose a new data selection method that exploits a diverse set of criteria that quantize interestingness of traffic scenes. Our experiments show that the proposed curation pipeline is able to select datasets that lead to better generalization and higher performance.
  arXiv  Detail & Related papers  (2021-01-16T23:45:02Z)
• Bayesian active learning for production, a systematic study and a reusable library [85.32971950095742]
  In this paper, we analyse the main drawbacks of current active learning techniques. We do a systematic study on the effects of the most common issues of real-world datasets on the deep active learning process. We derive two techniques that can speed up the active learning loop such as partial uncertainty sampling and larger query size.
  arXiv  Detail & Related papers  (2020-06-17T14:51:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.