Towards Robust and Reproducible Active Learning Using Neural Networks
- URL: http://arxiv.org/abs/2002.09564v3
- Date: Wed, 15 Jun 2022 18:50:54 GMT
- Title: Towards Robust and Reproducible Active Learning Using Neural Networks
- Authors: Prateek Munjal, Nasir Hayat, Munawar Hayat, Jamshid Sourati, Shadab
Khan
- Abstract summary: Active learning (AL) is a promising ML paradigm that has the potential to parse through large unlabeled data.
Recently proposed neural network based AL methods help reduce annotation cost in domains where labeling data can be prohibitive.
In this study, we demonstrate that different types of AL algorithms produce inconsistent gain over random sampling baseline.
- Score: 15.696979318409392
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Active learning (AL) is a promising ML paradigm that has the potential to
parse through large unlabeled data and help reduce annotation cost in domains
where labeling data can be prohibitive. Recently proposed neural network based
AL methods use different heuristics to accomplish this goal. In this study, we
demonstrate that under identical experimental settings, different types of AL
algorithms (uncertainty based, diversity based, and committee based) produce an
inconsistent gain over random sampling baseline. Through a variety of
experiments, controlling for sources of stochasticity, we show that variance in
performance metrics achieved by AL algorithms can lead to results that are not
consistent with the previously reported results. We also found that under
strong regularization, AL methods show marginal or no advantage over the random
sampling baseline under a variety of experimental conditions. Finally, we
conclude with a set of recommendations on how to assess the results using a new
AL algorithm to ensure results are reproducible and robust under changes in
experimental conditions. We share our codes to facilitate AL evaluations. We
believe our findings and recommendations will help advance reproducible
research in AL using neural networks. We open source our code at
https://github.com/PrateekMunjal/TorchAL
Related papers
- Unrolled denoising networks provably learn optimal Bayesian inference [54.79172096306631]
We prove the first rigorous learning guarantees for neural networks based on unrolling approximate message passing (AMP)
For compressed sensing, we prove that when trained on data drawn from a product prior, the layers of the network converge to the same denoisers used in Bayes AMP.
arXiv Detail & Related papers (2024-09-19T17:56:16Z) - Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples [53.95282502030541]
Neural Network-based active learning (NAL) is a cost-effective data selection technique that utilizes neural networks to select and train on a small subset of samples.
We try to move one step forward by offering a unified explanation for the success of both query criteria-based NAL from a feature learning view.
arXiv Detail & Related papers (2024-06-06T10:38:01Z) - Score-based Source Separation with Applications to Digital Communication
Signals [72.6570125649502]
We propose a new method for separating superimposed sources using diffusion-based generative models.
Motivated by applications in radio-frequency (RF) systems, we are interested in sources with underlying discrete nature.
Our method can be viewed as a multi-source extension to the recently proposed score distillation sampling scheme.
arXiv Detail & Related papers (2023-06-26T04:12:40Z) - On the Limitations of Simulating Active Learning [32.34440406689871]
Active learning (AL) is a human-and-model-in-the-loop paradigm that iteratively selects informative unlabeled data for human annotation.
An easy fix to this impediment is to simulate AL, by treating an already labeled and publicly available dataset as the pool of unlabeled data.
We argue that evaluating AL algorithms on available labeled datasets might provide a lower bound as to their effectiveness in real data.
arXiv Detail & Related papers (2023-05-21T22:52:13Z) - Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data.
We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
arXiv Detail & Related papers (2023-03-06T14:49:59Z) - Pareto Optimization for Active Learning under Out-of-Distribution Data
Scenarios [79.02009938011447]
We propose a sampling scheme, which selects optimal subsets of unlabeled samples with fixed batch size from the unlabeled data pool.
Experimental results show its effectiveness on both classical Machine Learning (ML) and Deep Learning (DL) tasks.
arXiv Detail & Related papers (2022-07-04T04:11:44Z) - Active Learning-Based Optimization of Scientific Experimental Design [1.9705094859539976]
Active learning (AL) is a machine learning algorithm that can achieve greater accuracy with fewer labeled training instances.
This article performs a retrospective study on a drug response dataset using the proposed AL scheme.
It shows that scientific experimental design, instead of being manually set, can be optimized by AL.
arXiv Detail & Related papers (2021-12-29T20:02:35Z) - Active learning for reducing labeling effort in text classification
tasks [3.8424737607413153]
Active learning (AL) is a paradigm that aims to reduce labeling effort by only using the data which the used model deems most informative.
We present an empirical study that compares different uncertainty-based algorithms BERT$_base$ as the used classifiers.
Our results show that using uncertainty-based AL with BERT$base$ outperforms random sampling of data.
arXiv Detail & Related papers (2021-09-10T13:00:36Z) - Local policy search with Bayesian optimization [73.0364959221845]
Reinforcement learning aims to find an optimal policy by interaction with an environment.
Policy gradients for local search are often obtained from random perturbations.
We develop an algorithm utilizing a probabilistic model of the objective function and its gradient.
arXiv Detail & Related papers (2021-06-22T16:07:02Z) - Active Structure Learning of Bayesian Networks in an Observational
Setting [21.376800678915558]
We study active structure learning of Bayesian networks in an observational setting.
We propose a new active learning algorithm that finds with a high probability a structure with a score that is $epsilon$-close to the optimal score.
We show that for a class of distributions that we term stable, a sample complexity reduction of up to a factor of $widetildeOmega(d3)$ can be obtained, where $d$ is the number of network variables.
arXiv Detail & Related papers (2021-03-25T12:50:14Z) - On the Robustness of Active Learning [0.7340017786387767]
Active Learning is concerned with how to identify the most useful samples for a Machine Learning algorithm to be trained with.
We find that it is often applied with not enough care and domain knowledge.
We propose the new "Sum of Squared Logits" method based on the Simpson diversity index and investigate the effect of using the confusion matrix for balancing in sample selection.
arXiv Detail & Related papers (2020-06-18T09:07:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.