Have we been Naive to Select Machine Learning Models? Noisy Data are here to Stay!

• URL: http://arxiv.org/abs/2207.06651v1
• Date: Thu, 14 Jul 2022 04:20:08 GMT
• Title: Have we been Naive to Select Machine Learning Models? Noisy Data are here to Stay!
• Authors: Felipe Costa Farias, Teresa Bernarda Ludermir and Carmelo Jos\'e Albanez Bastos-Filho
• Abstract summary: The model selection procedure is usually a single-criterion decision making in which we select the model that maximizes a specific metric in a specific set. We claim this is very naive and can perform poor selections of over-fitted models due to the over-searching phenomenon. We have defined four theoretical optimality conditions that we can pursue to better select the models and analyze them.
• Score: 2.094821665776961
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The model selection procedure is usually a single-criterion decision making in which we select the model that maximizes a specific metric in a specific set, such as the Validation set performance. We claim this is very naive and can perform poor selections of over-fitted models due to the over-searching phenomenon, which over-estimates the performance on that specific set. Futhermore, real world data contains noise that should not be ignored by the model selection procedure and must be taken into account when performing model selection. Also, we have defined four theoretical optimality conditions that we can pursue to better select the models and analyze them by using a multi-criteria decision-making algorithm (TOPSIS) that considers proxies to the optimality conditions to select reasonable models.

Related papers

• Stabilizing black-box model selection with the inflated argmax [8.52745154080651]
  This paper presents a new approach to stabilizing model selection that leverages a combination of bagging and an "inflated" argmax operation. Our method selects a small collection of models that all fit the data, and it is stable in that, with high probability, the removal of any training point will result in a collection of selected models that overlaps with the original collection. In both settings, the proposed method yields stable and compact collections of selected models, outperforming a variety of benchmarks.
  arXiv  Detail & Related papers  (2024-10-23T20:39:07Z)
• All models are wrong, some are useful: Model Selection with Limited Labels [49.62984196182567]
  We introduce MODEL SELECTOR, a framework for label-efficient selection of pretrained classifiers. We show that MODEL SELECTOR drastically reduces the need for labeled data while consistently picking the best or near-best performing model. Our results further highlight the robustness of MODEL SELECTOR in model selection, as it reduces the labeling cost by up to 72.41% when selecting a near-best model.
  arXiv  Detail & Related papers  (2024-10-17T14:45:56Z)
• An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting [53.36437745983783]
  We first construct a max-margin optimization-based model to model potentially non-monotonic preferences. We devise information amount measurement methods and question selection strategies to pinpoint the most informative alternative in each iteration. Two incremental preference elicitation-based algorithms are developed to learn potentially non-monotonic preferences.
  arXiv  Detail & Related papers  (2024-09-04T14:36:20Z)
• Towards Fundamentally Scalable Model Selection: Asymptotically Fast Update and Selection [40.85209520973634]
  An ideal model selection scheme should support two operations efficiently over a large pool of candidate models. Previous solutions to model selection require high computational complexity for at least one of these two operations. We present Standardized Embedder, an empirical realization of isolated model embedding.
  arXiv  Detail & Related papers  (2024-06-11T17:57:49Z)
• On the Laplace Approximation as Model Selection Criterion for Gaussian Processes [6.990493129893112]
  We introduce multiple metrics based on the Laplace approximation. Experiments show that our metrics are comparable in quality to the gold standard dynamic nested sampling.
  arXiv  Detail & Related papers  (2024-03-14T09:28:28Z)
• DsDm: Model-Aware Dataset Selection with Datamodels [81.01744199870043]
  Standard practice is to filter for examples that match human notions of data quality. We find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data. Our framework avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks.
  arXiv  Detail & Related papers  (2024-01-23T17:22:00Z)
• Budgeted Online Model Selection and Fine-Tuning via Federated Learning [26.823435733330705]
  Online model selection involves selecting a model from a set of candidate models 'on the fly' to perform prediction on a stream of data. The choice of candidate models henceforth has a crucial impact on the performance. The present paper proposes an online federated model selection framework where a group of learners (clients) interacts with a server with sufficient memory. Using the proposed algorithm, clients and the server collaborate to fine-tune models to adapt them to a non-stationary environment.
  arXiv  Detail & Related papers  (2024-01-19T04:02:49Z)
• Cost-Effective Online Contextual Model Selection [14.094350329970537]
  We formulate this task as an online contextual active model selection problem, where at each round the learner receives an unlabeled data point along with a context. The goal is to output the best model for any given context without obtaining an excessive amount of labels. We propose a contextual active model selection algorithm (CAMS), which relies on a novel uncertainty sampling query criterion defined on a given policy class for adaptive model selection.
  arXiv  Detail & Related papers  (2022-07-13T08:22:22Z)
• Modeling the Second Player in Distributionally Robust Optimization [90.25995710696425]
  We argue for the use of neural generative models to characterize the worst-case distribution. This approach poses a number of implementation and optimization challenges. We find that the proposed approach yields models that are more robust than comparable baselines.
  arXiv  Detail & Related papers  (2021-03-18T14:26:26Z)
• A linearized framework and a new benchmark for model selection for fine-tuning [112.20527122513668]
  Fine-tuning from a collection of models pre-trained on different domains is emerging as a technique to improve test accuracy in the low-data regime. We introduce two new baselines for model selection -- Label-Gradient and Label-Feature Correlation. Our benchmark highlights accuracy gain with model zoo compared to fine-tuning Imagenet models.
  arXiv  Detail & Related papers  (2021-01-29T21:57:15Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.