AutoML Benchmark with shorter time constraints and early stopping

• URL: http://arxiv.org/abs/2504.01222v3
• Date: Mon, 14 Apr 2025 23:15:34 GMT
• Title: AutoML Benchmark with shorter time constraints and early stopping
• Authors: Israel Campero Jurado, Pieter Gijsbers, Joaquin Vanschoren,
• Abstract summary: Automated Machine Learning (AutoML) automatically builds machine learning (ML) models on data.<n>The AutoML Benchmark (AMLB) proposed to evaluate AutoML frameworks using 1- and 4-hour time budgets across 104 tasks.<n>This work considers two ways in which to reduce the overall computation used in the benchmark: smaller time constraints and the use of early stopping.
• Score: 4.3513443677667585
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated Machine Learning (AutoML) automatically builds machine learning (ML) models on data. The de facto standard for evaluating new AutoML frameworks for tabular data is the AutoML Benchmark (AMLB). AMLB proposed to evaluate AutoML frameworks using 1- and 4-hour time budgets across 104 tasks. We argue that shorter time constraints should be considered for the benchmark because of their practical value, such as when models need to be retrained with high frequency, and to make AMLB more accessible. This work considers two ways in which to reduce the overall computation used in the benchmark: smaller time constraints and the use of early stopping. We conduct evaluations of 11 AutoML frameworks on 104 tasks with different time constraints and find the relative ranking of AutoML frameworks is fairly consistent across time constraints, but that using early-stopping leads to a greater variety in model performance.

Related papers

• UniAutoML: A Human-Centered Framework for Unified Discriminative and Generative AutoML with Large Language Models [5.725785427377439]
  We introduce UniAutoML, a human-centered AutoML framework that unifies AutoML for both discriminative and generative tasks. The human-centered design of UniAutoML innovatively features a conversational user interface (CUI) that facilitates natural language interactions. This design enhances transparency and user control throughout the AutoML training process, allowing users to seamlessly break down or modify the model being trained.
  arXiv  Detail & Related papers  (2024-10-09T17:33:15Z)
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline. Recent works have started exploiting large language models (LLM) to lessen such burden. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models [6.496539724366041]
  We introduce AutoM3L, an innovative Automated Multimodal Machine Learning framework. AutoM3L comprehends data modalities and selects appropriate models based on user requirements. We evaluate the performance of AutoM3L on six diverse multimodal datasets.
  arXiv  Detail & Related papers  (2024-08-01T16:01:51Z)
• AutoTimes: Autoregressive Time Series Forecasters via Large Language Models [67.83502953961505]
  AutoTimes projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. We formulate time series as prompts, extending the context for prediction beyond the lookback window. AutoTimes achieves state-of-the-art with 0.1% trainable parameters and over $5times$ training/inference speedup.
  arXiv  Detail & Related papers  (2024-02-04T06:59:21Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• AMLB: an AutoML Benchmark [9.642136611591578]
  We conduct a thorough comparison of 9 well-known AutoML frameworks across 71 classification and 33 regression tasks. The benchmark comes with an open-source tool that integrates with many AutoML frameworks and automates the empirical evaluation process end-to-end.
  arXiv  Detail & Related papers  (2022-07-25T22:34:08Z)
• Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning [45.643809726832764]
  We introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits. We also propose a solution towards truly hands-free AutoML.
  arXiv  Detail & Related papers  (2020-07-08T12:41:03Z)
• Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL [53.40030379661183]
  Auto-PyTorch is a framework to enable fully automated deep learning (AutoDL) It combines multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) We show that Auto-PyTorch performs better than several state-of-the-art competitors on average.
  arXiv  Detail & Related papers  (2020-06-24T15:15:17Z)
• Is deep learning necessary for simple classification tasks? [3.3793659640122717]
  Automated machine learning (AutoML) and deep learning (DL) are two cutting-edge paradigms used to solve inductive learning tasks. We compare AutoML and DL in the context of binary classification on 6 well-characterized public datasets. We also evaluate a new tool for genetic programming-based AutoML that incorporates deep estimators.
  arXiv  Detail & Related papers  (2020-06-11T18:41:47Z)
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data [120.2298620652828]
  We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models. Tests on a suite of 50 classification and regression tasks from Kaggle and the OpenML AutoML Benchmark reveal that AutoGluon is faster, more robust, and much more accurate.
  arXiv  Detail & Related papers  (2020-03-13T23:10:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.