Gaussian Switch Sampling: A Second Order Approach to Active Learning
- URL: http://arxiv.org/abs/2302.12018v1
- Date: Thu, 16 Feb 2023 15:24:56 GMT
- Title: Gaussian Switch Sampling: A Second Order Approach to Active Learning
- Authors: Ryan Benkert, Mohit Prabhushankar, Ghassan AlRegib, Armin Pacharmi,
and Enrique Corona
- Abstract summary: In active learning, acquisition functions define informativeness directly on the representation position within the model manifold.
We propose a grounded second-order definition of information content and sample importance within the context of active learning.
We show that our definition produces highly accurate importance scores even when the model representations are constrained by the lack of training data.
- Score: 11.775252660867285
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In active learning, acquisition functions define informativeness directly on
the representation position within the model manifold. However, for most
machine learning models (in particular neural networks) this representation is
not fixed due to the training pool fluctuations in between active learning
rounds. Therefore, several popular strategies are sensitive to experiment
parameters (e.g. architecture) and do not consider model robustness to
out-of-distribution settings. To alleviate this issue, we propose a grounded
second-order definition of information content and sample importance within the
context of active learning. Specifically, we define importance by how often a
neural network "forgets" a sample during training - artifacts of second order
representation shifts. We show that our definition produces highly accurate
importance scores even when the model representations are constrained by the
lack of training data. Motivated by our analysis, we develop Gaussian Switch
Sampling (GauSS). We show that GauSS is setup agnostic and robust to anomalous
distributions with exhaustive experiments on three in-distribution benchmarks,
three out-of-distribution benchmarks, and three different architectures. We
report an improvement of up to 5% when compared against four popular query
strategies.
Related papers
- Learning Active Subspaces and Discovering Important Features with Gaussian Radial Basis Functions Neural Networks [0.0]
We show that precious information is contained in the spectrum of the precision matrix that can be extracted once the training of the model is completed.
We conducted numerical experiments for regression, classification, and feature selection tasks.
Our results demonstrate that the proposed model does not only yield an attractive prediction performance compared to the competitors.
arXiv Detail & Related papers (2023-07-11T09:54:30Z) - Robust Outlier Rejection for 3D Registration with Variational Bayes [70.98659381852787]
We develop a novel variational non-local network-based outlier rejection framework for robust alignment.
We propose a voting-based inlier searching strategy to cluster the high-quality hypothetical inliers for transformation estimation.
arXiv Detail & Related papers (2023-04-04T03:48:56Z) - Self-supervised learning of Split Invariant Equivariant representations [0.0]
We introduce 3DIEBench, consisting of renderings from 3D models over 55 classes and more than 2.5 million images where we have full control on the transformations applied to the objects.
We introduce a predictor architecture based on hypernetworks to learn equivariant representations with no possible collapse to invariance.
We introduce SIE (Split Invariant-Equivariant) which combines the hypernetwork-based predictor with representations split in two parts, one invariant, the other equivariant, to learn richer representations.
arXiv Detail & Related papers (2023-02-14T07:53:18Z) - Forgetful Active Learning with Switch Events: Efficient Sampling for
Out-of-Distribution Data [13.800680101300756]
In practice, fully trained neural networks interact randomly with out-of-distribution (OOD) inputs.
We introduce forgetful active learning with switch events (FALSE) - a novel active learning protocol for out-of-distribution active learning.
We report up to 4.5% accuracy improvements in over 270 experiments.
arXiv Detail & Related papers (2023-01-12T16:03:14Z) - CHALLENGER: Training with Attribution Maps [63.736435657236505]
We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance.
In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
arXiv Detail & Related papers (2022-05-30T13:34:46Z) - GDC- Generalized Distribution Calibration for Few-Shot Learning [5.076419064097734]
Few shot learning is an important problem in machine learning as large labelled datasets take considerable time and effort to assemble.
Most few-shot learning algorithms suffer from one of two limitations- they either require the design of sophisticated models and loss functions, thus hampering interpretability.
We propose a Generalized sampling method that learns to estimate few-shot distributions for classification as weighted random variables of all large classes.
arXiv Detail & Related papers (2022-04-11T16:22:53Z) - Towards Open-World Feature Extrapolation: An Inductive Graph Learning
Approach [80.8446673089281]
We propose a new learning paradigm with graph representation and learning.
Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
arXiv Detail & Related papers (2021-10-09T09:02:45Z) - Deep Active Learning in Remote Sensing for data efficient Change
Detection [26.136331738529243]
We investigate active learning in the context of deep neural network models for change detection and map updating.
In active learning, one starts from a minimal set of training examples and progressively chooses informative samples annotated by a user.
We show that active learning successfully finds highly informative samples and automatically balances the training distribution.
arXiv Detail & Related papers (2020-08-25T17:58:17Z) - Automatic Recall Machines: Internal Replay, Continual Learning and the
Brain [104.38824285741248]
Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity.
We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective.
Instead the implicit memory of learned samples within the assessed model itself is exploited.
arXiv Detail & Related papers (2020-06-22T15:07:06Z) - Learning What Makes a Difference from Counterfactual Examples and
Gradient Supervision [57.14468881854616]
We propose an auxiliary training objective that improves the generalization capabilities of neural networks.
We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task.
Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
arXiv Detail & Related papers (2020-04-20T02:47:49Z) - Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction.
We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data.
Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
arXiv Detail & Related papers (2020-03-10T03:10:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.