Beyond Active Learning: Leveraging the Full Potential of Human
Interaction via Auto-Labeling, Human Correction, and Human Verification
- URL: http://arxiv.org/abs/2306.01277v1
- Date: Fri, 2 Jun 2023 05:40:11 GMT
- Title: Beyond Active Learning: Leveraging the Full Potential of Human
Interaction via Auto-Labeling, Human Correction, and Human Verification
- Authors: Nathan Beck, Krishnateja Killamsetty, Suraj Kothawade, Rishabh Iyer
- Abstract summary: Active Learning (AL) is a human-in-the-loop framework to interactively and adaptively label data instances.
We propose CLARIFIER, an Interactive Learning framework that admits more effective use of human interaction by leveraging the reduced cost of verification.
- Score: 3.58439716487063
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Active Learning (AL) is a human-in-the-loop framework to interactively and
adaptively label data instances, thereby enabling significant gains in model
performance compared to random sampling. AL approaches function by selecting
the hardest instances to label, often relying on notions of diversity and
uncertainty. However, we believe that these current paradigms of AL do not
leverage the full potential of human interaction granted by automated label
suggestions. Indeed, we show that for many classification tasks and datasets,
most people verifying if an automatically suggested label is correct take
$3\times$ to $4\times$ less time than they do changing an incorrect suggestion
to the correct label (or labeling from scratch without any suggestion).
Utilizing this result, we propose CLARIFIER (aCtive LeARnIng From tIEred
haRdness), an Interactive Learning framework that admits more effective use of
human interaction by leveraging the reduced cost of verification. By targeting
the hard (uncertain) instances with existing AL methods, the intermediate
instances with a novel label suggestion scheme using submodular mutual
information functions on a per-class basis, and the easy (confident) instances
with highest-confidence auto-labeling, CLARIFIER can improve over the
performance of existing AL approaches on multiple datasets -- particularly on
those that have a large number of classes -- by almost 1.5$\times$ to 2$\times$
in terms of relative labeling cost.
Related papers
- Federated Learning with Only Positive Labels by Exploring Label Correlations [78.59613150221597]
Federated learning aims to collaboratively learn a model by using the data from multiple users under privacy constraints.
In this paper, we study the multi-label classification problem under the federated learning setting.
We propose a novel and generic method termed Federated Averaging by exploring Label Correlations (FedALC)
arXiv Detail & Related papers (2024-04-24T02:22:50Z) - Improving Classifier Robustness through Active Generation of Pairwise
Counterfactuals [22.916599410472102]
We present a novel framework that utilizes counterfactual generative models to generate a large number of diverse counterfactuals.
We show that with a small amount of human-annotated counterfactual data (10%), we can generate a counterfactual augmentation dataset with learned labels.
arXiv Detail & Related papers (2023-05-22T23:19:01Z) - Eliciting and Learning with Soft Labels from Every Annotator [31.10635260890126]
We focus on efficiently eliciting soft labels from individual annotators.
We demonstrate that learning with our labels achieves comparable model performance to prior approaches.
arXiv Detail & Related papers (2022-07-02T12:03:00Z) - One Positive Label is Sufficient: Single-Positive Multi-Label Learning
with Label Enhancement [71.9401831465908]
We investigate single-positive multi-label learning (SPMLL) where each example is annotated with only one relevant label.
A novel method named proposed, i.e., Single-positive MultI-label learning with Label Enhancement, is proposed.
Experiments on benchmark datasets validate the effectiveness of the proposed method.
arXiv Detail & Related papers (2022-06-01T14:26:30Z) - A Lagrangian Duality Approach to Active Learning [119.36233726867992]
We consider the batch active learning problem, where only a subset of the training data is labeled.
We formulate the learning problem using constrained optimization, where each constraint bounds the performance of the model on labeled samples.
We show, via numerical experiments, that our proposed approach performs similarly to or better than state-of-the-art active learning methods.
arXiv Detail & Related papers (2022-02-08T19:18:49Z) - Cost-Accuracy Aware Adaptive Labeling for Active Learning [9.761953860259942]
In many real settings, different labelers have different labeling costs and can yield different labeling accuracies.
We propose a new algorithm for selecting instances, labelers and their corresponding costs and labeling accuracies.
Our proposed algorithm demonstrates state-of-the-art performance on five UCI and a real crowdsourcing dataset.
arXiv Detail & Related papers (2021-05-24T17:21:00Z) - Group-aware Label Transfer for Domain Adaptive Person Re-identification [179.816105255584]
Unsupervised Adaptive Domain (UDA) person re-identification (ReID) aims at adapting the model trained on a labeled source-domain dataset to a target-domain dataset without any further annotations.
Most successful UDA-ReID approaches combine clustering-based pseudo-label prediction with representation learning and perform the two steps in an alternating fashion.
We propose a Group-aware Label Transfer (GLT) algorithm, which enables the online interaction and mutual promotion of pseudo-label prediction and representation learning.
arXiv Detail & Related papers (2021-03-23T07:57:39Z) - Active Domain Adaptation via Clustering Uncertainty-weighted Embeddings [57.24806659432956]
Generalizing deep neural networks to new target domains is critical to their real-world utility.
We study the problem of active learning (AL) under a domain shift, called Active Domain Adaptation (Active DA)
We propose Clustering Uncertainty-weighted Embeddings (CLUE), a novel label acquisition strategy for Active DA that performs uncertainty-weighted clustering to identify target instances for labeling that are both uncertain under the model and diverse in feature space.
arXiv Detail & Related papers (2020-10-16T23:37:44Z) - Interaction Matching for Long-Tail Multi-Label Classification [57.262792333593644]
We present an elegant and effective approach for addressing limitations in existing multi-label classification models.
By performing soft n-gram interaction matching, we match labels with natural language descriptions.
arXiv Detail & Related papers (2020-05-18T15:27:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.