Cost-Accuracy Aware Adaptive Labeling for Active Learning
- URL: http://arxiv.org/abs/2105.11418v1
- Date: Mon, 24 May 2021 17:21:00 GMT
- Title: Cost-Accuracy Aware Adaptive Labeling for Active Learning
- Authors: Ruijiang Gao, Maytal Saar-tsechansky
- Abstract summary: In many real settings, different labelers have different labeling costs and can yield different labeling accuracies.
We propose a new algorithm for selecting instances, labelers and their corresponding costs and labeling accuracies.
Our proposed algorithm demonstrates state-of-the-art performance on five UCI and a real crowdsourcing dataset.
- Score: 9.761953860259942
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Conventional active learning algorithms assume a single labeler that produces
noiseless label at a given, fixed cost, and aim to achieve the best
generalization performance for given classifier under a budget constraint.
However, in many real settings, different labelers have different labeling
costs and can yield different labeling accuracies. Moreover, a given labeler
may exhibit different labeling accuracies for different instances. This setting
can be referred to as active learning with diverse labelers with varying costs
and accuracies, and it arises in many important real settings. It is therefore
beneficial to understand how to effectively trade-off between labeling accuracy
for different instances, labeling costs, as well as the informativeness of
training instances, so as to achieve the best generalization performance at the
lowest labeling cost. In this paper, we propose a new algorithm for selecting
instances, labelers (and their corresponding costs and labeling accuracies),
that employs generalization bound of learning with label noise to select
informative instances and labelers so as to achieve higher generalization
accuracy at a lower cost. Our proposed algorithm demonstrates state-of-the-art
performance on five UCI and a real crowdsourcing dataset.
Related papers
- Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations [91.67511167969934]
imprecise label learning (ILL) is a framework for the unification of learning with various imprecise label configurations.
We demonstrate that ILL can seamlessly adapt to partial label learning, semi-supervised learning, noisy label learning, and, more importantly, a mixture of these settings.
arXiv Detail & Related papers (2023-05-22T04:50:28Z) - Improved Adaptive Algorithm for Scalable Active Learning with Weak
Labeler [89.27610526884496]
Weak Labeler Active Cover (WL-AC) is able to robustly leverage the lower quality weak labelers to reduce the query complexity while retaining the desired level of accuracy.
We show its effectiveness on the corrupted-MNIST dataset by significantly reducing the number of labels while keeping the same accuracy as in passive learning.
arXiv Detail & Related papers (2022-11-04T02:52:54Z) - Eliciting and Learning with Soft Labels from Every Annotator [31.10635260890126]
We focus on efficiently eliciting soft labels from individual annotators.
We demonstrate that learning with our labels achieves comparable model performance to prior approaches.
arXiv Detail & Related papers (2022-07-02T12:03:00Z) - Debiased Pseudo Labeling in Self-Training [77.83549261035277]
Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets.
To mitigate the requirement for labeled data, self-training is widely used in both academia and industry by pseudo labeling on readily-available unlabeled data.
We propose Debiased, in which the generation and utilization of pseudo labels are decoupled by two independent heads.
arXiv Detail & Related papers (2022-02-15T02:14:33Z) - Learning with Proper Partial Labels [87.65718705642819]
Partial-label learning is a kind of weakly-supervised learning with inexact labels.
We show that this proper partial-label learning framework includes many previous partial-label learning settings.
We then derive a unified unbiased estimator of the classification risk.
arXiv Detail & Related papers (2021-12-23T01:37:03Z) - Learning with Noisy Labels by Targeted Relabeling [52.0329205268734]
Crowdsourcing platforms are often used to collect datasets for training deep neural networks.
We propose an approach which reserves a fraction of annotations to explicitly relabel highly probable labeling errors.
arXiv Detail & Related papers (2021-10-15T20:37:29Z) - A Study on the Autoregressive and non-Autoregressive Multi-label
Learning [77.11075863067131]
We propose a self-attention based variational encoder-model to extract the label-label and label-feature dependencies jointly.
Our model can therefore be used to predict all labels in parallel while still including both label-label and label-feature dependencies.
arXiv Detail & Related papers (2020-12-03T05:41:44Z) - Active Learning for Noisy Data Streams Using Weak and Strong Labelers [3.9370369973510746]
We consider a novel weak and strong labeler problem inspired by humans natural ability for labeling.
We propose an on-line active learning algorithm that consists of four steps: filtering, adding diversity, informative sample selection, and labeler selection.
We derive a decision function that measures the information gain by combining the informativeness of individual samples and model confidence.
arXiv Detail & Related papers (2020-10-27T09:18:35Z) - Cost-Quality Adaptive Active Learning for Chinese Clinical Named Entity
Recognition [4.227856561940623]
We propose a Cost-Quality Adaptive Active Learning (CQAAL) approach for Clinical Named Entity Recognition (CNER) in Chinese EHRs.
CQAAL selects cost-effective instance-labeler pairs to achieve better annotation quality with lower costs in an adaptive manner.
arXiv Detail & Related papers (2020-08-28T09:27:43Z) - Minimum Cost Active Labeling [2.0754848504005587]
min-cost labeling uses a variant of active learning to learn a model to predict the optimal training set size.
In some cases, our approach has 6X lower overall cost relative to human labeling, and is always cheaper than the cheapest active learning strategy.
arXiv Detail & Related papers (2020-06-24T19:01:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.