Learning active learning at the crossroads? evaluation and discussion
- URL: http://arxiv.org/abs/2012.09631v1
- Date: Wed, 16 Dec 2020 10:35:43 GMT
- Title: Learning active learning at the crossroads? evaluation and discussion
- Authors: Louis Desreumaux and Vincent Lemaire
- Abstract summary: Active learning aims to reduce annotation cost by predicting which samples are useful for a human expert to label.
There is no best active learning strategy that consistently outperforms all others in all applications.
We present the results of a benchmark performed on 20 datasets that compares a strategy learned using a recent meta-learning algorithm with margin sampling.
- Score: 0.03807314298073299
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Active learning aims to reduce annotation cost by predicting which samples
are useful for a human expert to label. Although this field is quite old,
several important challenges to using active learning in real-world settings
still remain unsolved. In particular, most selection strategies are
hand-designed, and it has become clear that there is no best active learning
strategy that consistently outperforms all others in all applications. This has
motivated research into meta-learning algorithms for "learning how to actively
learn". In this paper, we compare this kind of approach with the association of
a Random Forest with the margin sampling strategy, reported in recent
comparative studies as a very competitive heuristic. To this end, we present
the results of a benchmark performed on 20 datasets that compares a strategy
learned using a recent meta-learning algorithm with margin sampling. We also
present some lessons learned and open future perspectives.
Related papers
- ALPBench: A Benchmark for Active Learning Pipelines on Tabular Data [18.553222868627792]
In settings where only a budgeted amount of labeled data can be afforded, active learning seeks to devise query strategies for selecting the most informative data points to be labeled.
Numerous such query strategies have been proposed and compared in the active learning literature.
The community still lacks standardized benchmarks for comparing the performance of different query strategies.
arXiv Detail & Related papers (2024-06-25T07:14:14Z) - BAL: Balancing Diversity and Novelty for Active Learning [53.289700543331925]
We introduce a novel framework, Balancing Active Learning (BAL), which constructs adaptive sub-pools to balance diverse and uncertain data.
Our approach outperforms all established active learning methods on widely recognized benchmarks by 1.20%.
arXiv Detail & Related papers (2023-12-26T08:14:46Z) - ALE: A Simulation-Based Active Learning Evaluation Framework for the
Parameter-Driven Comparison of Query Strategies for NLP [3.024761040393842]
Active Learning (AL) proposes promising data points to annotators they annotate next instead of a subsequent or random sample.
This method is supposed to save annotation effort while maintaining model performance.
We introduce a reproducible active learning evaluation framework for the comparative evaluation of AL strategies in NLP.
arXiv Detail & Related papers (2023-08-01T10:42:11Z) - Algorithm Selection for Deep Active Learning with Imbalanced Datasets [11.902019233549474]
Active learning aims to reduce the number of labeled examples needed to train deep networks.
It is difficult to know in advance which active learning strategy will perform well or best in a given application.
We propose the first adaptive algorithm selection strategy for deep active learning.
arXiv Detail & Related papers (2023-02-14T19:59:49Z) - What Makes Good Contrastive Learning on Small-Scale Wearable-based
Tasks? [59.51457877578138]
We study contrastive learning on the wearable-based activity recognition task.
This paper presents an open-source PyTorch library textttCL-HAR, which can serve as a practical tool for researchers.
arXiv Detail & Related papers (2022-02-12T06:10:15Z) - Budget-aware Few-shot Learning via Graph Convolutional Network [56.41899553037247]
This paper tackles the problem of few-shot learning, which aims to learn new visual concepts from a few examples.
A common problem setting in few-shot classification assumes random sampling strategy in acquiring data labels.
We introduce a new budget-aware few-shot learning problem that aims to learn novel object categories.
arXiv Detail & Related papers (2022-01-07T02:46:35Z) - CoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised
Learning [58.107679606345165]
Noisy label learning, semi-supervised learning, and contrastive learning are three different strategies for designing learning processes requiring less annotation cost.
We propose CSSL, a unified Contrastive Semi-Supervised Learning algorithm, and CoDiM, a novel algorithm for learning with noisy labels.
arXiv Detail & Related papers (2021-11-23T04:56:40Z) - Meta Navigator: Search for a Good Adaptation Policy for Few-shot
Learning [113.05118113697111]
Few-shot learning aims to adapt knowledge learned from previous tasks to novel tasks with only a limited amount of labeled data.
Research literature on few-shot learning exhibits great diversity, while different algorithms often excel at different few-shot learning scenarios.
We present Meta Navigator, a framework that attempts to solve the limitation in few-shot learning by seeking a higher-level strategy.
arXiv Detail & Related papers (2021-09-13T07:20:01Z) - Rebuilding Trust in Active Learning with Actionable Metrics [77.99796068970569]
Active Learning (AL) is an active domain of research, but is seldom used in the industry despite the pressing needs.
This is in part due to a misalignment of objectives, while research strives at getting the best results on selected datasets.
We present various actionable metrics to help rebuild trust of industrial practitioners in Active Learning.
arXiv Detail & Related papers (2020-12-18T09:34:59Z) - Active Output Selection Strategies for Multiple Learning Regression
Models [0.0]
The strategy is actively learning multiple outputs in the same input space.
The presented method is applied to three different toy examples with noise in a real world range and to a benchmark dataset.
The results are promising but also show that the algorithm has to be improved to increase robustness for noisy environments.
arXiv Detail & Related papers (2020-11-29T08:05:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.