Can AutoML outperform humans? An evaluation on popular OpenML datasets using AutoML Benchmark

• URL: http://arxiv.org/abs/2009.01564v2
• Date: Tue, 15 Dec 2020 09:33:02 GMT
• Title: Can AutoML outperform humans? An evaluation on popular OpenML datasets using AutoML Benchmark
• Authors: Marc Hanussek, Matthias Blohm, Maximilien Kintz
• Abstract summary: This paper compares four AutoML frameworks on 12 different popular datasets from OpenML. Results show that the automated frameworks perform better or equal than the machine learning community in 7 out of 12 OpenML tasks.
• Score: 0.05156484100374058
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the last few years, Automated Machine Learning (AutoML) has gained much attention. With that said, the question arises whether AutoML can outperform results achieved by human data scientists. This paper compares four AutoML frameworks on 12 different popular datasets from OpenML; six of them supervised classification tasks and the other six supervised regression ones. Additionally, we consider a real-life dataset from one of our recent projects. The results show that the automated frameworks perform better or equal than the machine learning community in 7 out of 12 OpenML tasks.

Related papers

• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline. Recent works have started exploiting large language models (LLM) to lessen such burden. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• Position: A Call to Action for a Human-Centered AutoML Paradigm [83.78883610871867]
  Automated machine learning (AutoML) was formed around the fundamental objectives of automatically and efficiently configuring machine learning (ML) We argue that a key to unlocking AutoML's full potential lies in addressing the currently underexplored aspect of user interaction with AutoML systems.
  arXiv  Detail & Related papers  (2024-06-05T15:05:24Z)
• The Potential of AutoML for Recommender Systems [0.135975510645475]
  We compare the predictive performance of 60 AutoML, AutoRecSys, ML, and RecSys algorithms from 15 libraries. We found that AutoML and AutoRecSys libraries performed best for six of the 14 datasets. The single-best library was the AutoRecSys library Auto-Surprise, which performed best on five datasets.
  arXiv  Detail & Related papers  (2024-02-06T22:42:28Z)
• MLLM-DataEngine: An Iterative Refinement Approach for MLLM [62.30753425449056]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data. For quality, we resort to GPT-4 to generate high-quality data with each given data type.
  arXiv  Detail & Related papers  (2023-08-25T01:41:04Z)
• Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering [52.09178018466104]
  We introduce Context-Aware Automated Feature Engineering (CAAFE) to generate semantically meaningful features for datasets. Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets. We highlight the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML.
  arXiv  Detail & Related papers  (2023-05-05T09:58:40Z)
• AMLB: an AutoML Benchmark [9.642136611591578]
  We conduct a thorough comparison of 9 well-known AutoML frameworks across 71 classification and 33 regression tasks. The benchmark comes with an open-source tool that integrates with many AutoML frameworks and automates the empirical evaluation process end-to-end.
  arXiv  Detail & Related papers  (2022-07-25T22:34:08Z)
• Leveraging Automated Machine Learning for Text Classification: Evaluation of AutoML Tools and Comparison with Human Performance [0.07734726150561087]
  This work compares four AutoML tools on 13 different popular datasets. Results show that the AutoML tools perform better than the machine learning community in 4 out of 13 tasks.
  arXiv  Detail & Related papers  (2020-12-07T10:31:13Z)
• Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning [45.643809726832764]
  We introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits. We also propose a solution towards truly hands-free AutoML.
  arXiv  Detail & Related papers  (2020-07-08T12:41:03Z)
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data [120.2298620652828]
  We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models. Tests on a suite of 50 classification and regression tasks from Kaggle and the OpenML AutoML Benchmark reveal that AutoGluon is faster, more robust, and much more accurate.
  arXiv  Detail & Related papers  (2020-03-13T23:10:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.