Cost-Effective Online Contextual Model Selection
- URL: http://arxiv.org/abs/2207.06030v1
- Date: Wed, 13 Jul 2022 08:22:22 GMT
- Title: Cost-Effective Online Contextual Model Selection
- Authors: Xuefeng Liu, Fangfang Xia, Rick L. Stevens, Yuxin Chen
- Abstract summary: We formulate this task as an online contextual active model selection problem, where at each round the learner receives an unlabeled data point along with a context.
The goal is to output the best model for any given context without obtaining an excessive amount of labels.
We propose a contextual active model selection algorithm (CAMS), which relies on a novel uncertainty sampling query criterion defined on a given policy class for adaptive model selection.
- Score: 14.094350329970537
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: How can we collect the most useful labels to learn a model selection policy,
when presented with arbitrary heterogeneous data streams? In this paper, we
formulate this task as an online contextual active model selection problem,
where at each round the learner receives an unlabeled data point along with a
context. The goal is to output the best model for any given context without
obtaining an excessive amount of labels. In particular, we focus on the task of
selecting pre-trained classifiers, and propose a contextual active model
selection algorithm (CAMS), which relies on a novel uncertainty sampling query
criterion defined on a given policy class for adaptive model selection. In
comparison to prior art, our algorithm does not assume a globally optimal
model. We provide rigorous theoretical analysis for the regret and query
complexity under both adversarial and stochastic settings. Our experiments on
several benchmark classification datasets demonstrate the algorithm's
effectiveness in terms of both regret and query complexity. Notably, to achieve
the same accuracy, CAMS incurs less than 10% of the label cost when compared to
the best online model selection baselines on CIFAR10.
Related papers
- An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting [53.36437745983783]
We first construct a max-margin optimization-based model to model potentially non-monotonic preferences.
We devise information amount measurement methods and question selection strategies to pinpoint the most informative alternative in each iteration.
Two incremental preference elicitation-based algorithms are developed to learn potentially non-monotonic preferences.
arXiv Detail & Related papers (2024-09-04T14:36:20Z) - DsDm: Model-Aware Dataset Selection with Datamodels [81.01744199870043]
Standard practice is to filter for examples that match human notions of data quality.
We find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data.
Our framework avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks.
arXiv Detail & Related papers (2024-01-23T17:22:00Z) - Towards Free Data Selection with General-Purpose Models [71.92151210413374]
A desirable data selection algorithm can efficiently choose the most informative samples to maximize the utility of limited annotation budgets.
Current approaches, represented by active learning methods, typically follow a cumbersome pipeline that iterates the time-consuming model training and batch data selection repeatedly.
FreeSel bypasses the heavy batch selection process, achieving a significant improvement in efficiency and being 530x faster than existing active learning methods.
arXiv Detail & Related papers (2023-09-29T15:50:14Z) - A Lagrangian Duality Approach to Active Learning [119.36233726867992]
We consider the batch active learning problem, where only a subset of the training data is labeled.
We formulate the learning problem using constrained optimization, where each constraint bounds the performance of the model on labeled samples.
We show, via numerical experiments, that our proposed approach performs similarly to or better than state-of-the-art active learning methods.
arXiv Detail & Related papers (2022-02-08T19:18:49Z) - Consistent Relative Confidence and Label-Free Model Selection for
Convolutional Neural Networks [4.497097230665825]
This paper presents an approach to CNN model selection using only unlabeled data.
The effectiveness and efficiency of the presented method are demonstrated by extensive experimental studies based on datasets MNIST and FasionMNIST.
arXiv Detail & Related papers (2021-08-26T15:14:38Z) - Multiple-criteria Based Active Learning with Fixed-size Determinantal
Point Processes [43.71112693633952]
We introduce a multiple-criteria based active learning algorithm, which incorporates three complementary criteria, i.e., informativeness, representativeness and diversity.
We show that our method performs significantly better and is more stable than other multiple-criteria based AL algorithms.
arXiv Detail & Related papers (2021-07-04T13:22:54Z) - Adaptive Threshold for Better Performance of the Recognition and
Re-identification Models [0.0]
An online optimization-based statistical feature learning adaptive technique is developed and tested on the LFW datasets and self-prepared athletes datasets.
This method of adopting adaptive threshold resulted in 12-45% improvement in the model accuracy compared to the fixed threshold 0.3,0.5,0.7 that are usually taken via the hit-and-trial method in any classification and identification tasks.
arXiv Detail & Related papers (2020-12-28T15:40:53Z) - Online Active Model Selection for Pre-trained Classifiers [72.84853880948894]
We design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round.
Our algorithm can be used for online prediction tasks for both adversarial and streams.
arXiv Detail & Related papers (2020-10-19T19:53:15Z) - Progressive Identification of True Labels for Partial-Label Learning [112.94467491335611]
Partial-label learning (PLL) is a typical weakly supervised learning problem, where each training instance is equipped with a set of candidate labels among which only one is the true label.
Most existing methods elaborately designed as constrained optimizations that must be solved in specific manners, making their computational complexity a bottleneck for scaling up to big data.
This paper proposes a novel framework of classifier with flexibility on the model and optimization algorithm.
arXiv Detail & Related papers (2020-02-19T08:35:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.