Contextual Active Model Selection

• URL: http://arxiv.org/abs/2207.06030v4
• Date: Sun, 09 Feb 2025 12:19:57 GMT
• Title: Contextual Active Model Selection
• Authors: Xuefeng Liu, Fangfang Xia, Rick L. Stevens, Yuxin Chen,
• Abstract summary: We present an approach to actively select pre-trained models while minimizing labeling costs. The objective is to adaptively select the best model to make a prediction while limiting label requests. We propose CAMS, a contextual active model selection algorithm that relies on two novel components.
• Score: 10.925932167673764
• License:
• Abstract: While training models and labeling data are resource-intensive, a wealth of pre-trained models and unlabeled data exists. To effectively utilize these resources, we present an approach to actively select pre-trained models while minimizing labeling costs. We frame this as an online contextual active model selection problem: At each round, the learner receives an unlabeled data point as a context. The objective is to adaptively select the best model to make a prediction while limiting label requests. To tackle this problem, we propose CAMS, a contextual active model selection algorithm that relies on two novel components: (1) a contextual model selection mechanism, which leverages context information to make informed decisions about which model is likely to perform best for a given context, and (2) an active query component, which strategically chooses when to request labels for data points, minimizing the overall labeling cost. We provide rigorous theoretical analysis for the regret and query complexity under both adversarial and stochastic settings. Furthermore, we demonstrate the effectiveness of our algorithm on a diverse collection of benchmark classification tasks. Notably, CAMS requires substantially less labeling effort (less than 10%) compared to existing methods on CIFAR10 and DRIFT benchmarks, while achieving similar or better accuracy. Our code is publicly available at: https://github.com/xuefeng-cs/Contextual-Active-Model-Selection.

Related papers

• Enabling Small Models for Zero-Shot Selection and Reuse through Model Label Learning [50.68074833512999]
  We introduce a novel paradigm, Model Label Learning (MLL), which bridges the gap between models and their functionalities. Experiments on seven real-world datasets validate the effectiveness and efficiency of MLL.
  arXiv  Detail & Related papers  (2024-08-21T09:08:26Z)
• Jump-teaching: Ultra Efficient and Robust Learning with Noisy Label [6.818488262543482]
  We propose a novel technique to distinguish mislabeled samples during training. We employ only one network with the jump manner update to decouple the interplay and mine more semantic information from the loss. Our proposed approach achieves almost up to $2.53times$ speedup, $0.56times$ peak memory footprint, and superior robustness over state-of-the-art works with various noise settings.
  arXiv  Detail & Related papers  (2024-05-27T12:54:09Z)
• DsDm: Model-Aware Dataset Selection with Datamodels [81.01744199870043]
  Standard practice is to filter for examples that match human notions of data quality. We find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data. Our framework avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks.
  arXiv  Detail & Related papers  (2024-01-23T17:22:00Z)
• How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking [40.39898960460575]
  This paper presents LEMR (Label-Efficient Model Ranking) and introduces the MoraBench Benchmark. LEMR is a novel framework that minimizes the need for costly annotations in model selection by strategically annotating instances from an unlabeled validation set.
  arXiv  Detail & Related papers  (2023-12-04T04:20:38Z)
• GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks [3.9638110494107095]
  In-context Learning (ICL) is the ability of Large Language Models (LLMs) to perform new tasks when conditioned on prompts. We propose Example Gisting, a novel approach for training example encoders through supervised fine-tuning. We show that our fine-tuned models get state-of-the-art ICL performance with over 20% absolute gain over off-the-shelf retrievers.
  arXiv  Detail & Related papers  (2023-11-16T06:28:05Z)
• Ground Truth Inference for Weakly Supervised Entity Matching [76.6732856489872]
  We propose a simple but powerful labeling model for weak supervision tasks. We then tailor the labeling model specifically to the task of entity matching. We show that our labeling model results in a 9% higher F1 score on average than the best existing method.
  arXiv  Detail & Related papers  (2022-11-13T17:57:07Z)
• Active metric learning and classification using similarity queries [21.589707834542338]
  We show that a novel unified query framework can be applied to any problem in which a key component is learning a representation of the data that reflects similarity. We demonstrate the effectiveness of the proposed strategy on two tasks -- active metric learning and active classification.
  arXiv  Detail & Related papers  (2022-02-04T03:34:29Z)
• Multiple-criteria Based Active Learning with Fixed-size Determinantal Point Processes [43.71112693633952]
  We introduce a multiple-criteria based active learning algorithm, which incorporates three complementary criteria, i.e., informativeness, representativeness and diversity. We show that our method performs significantly better and is more stable than other multiple-criteria based AL algorithms.
  arXiv  Detail & Related papers  (2021-07-04T13:22:54Z)
• A linearized framework and a new benchmark for model selection for fine-tuning [112.20527122513668]
  Fine-tuning from a collection of models pre-trained on different domains is emerging as a technique to improve test accuracy in the low-data regime. We introduce two new baselines for model selection -- Label-Gradient and Label-Feature Correlation. Our benchmark highlights accuracy gain with model zoo compared to fine-tuning Imagenet models.
  arXiv  Detail & Related papers  (2021-01-29T21:57:15Z)
• Online Active Model Selection for Pre-trained Classifiers [72.84853880948894]
  We design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can be used for online prediction tasks for both adversarial and streams.
  arXiv  Detail & Related papers  (2020-10-19T19:53:15Z)
• Progressive Identification of True Labels for Partial-Label Learning [112.94467491335611]
  Partial-label learning (PLL) is a typical weakly supervised learning problem, where each training instance is equipped with a set of candidate labels among which only one is the true label. Most existing methods elaborately designed as constrained optimizations that must be solved in specific manners, making their computational complexity a bottleneck for scaling up to big data. This paper proposes a novel framework of classifier with flexibility on the model and optimization algorithm.
  arXiv  Detail & Related papers  (2020-02-19T08:35:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.