Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

• URL: http://arxiv.org/abs/2006.13799v3
• Date: Mon, 26 Apr 2021 10:36:34 GMT
• Title: Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
• Authors: Lucas Zimmer, Marius Lindauer, Frank Hutter
• Abstract summary: Auto-PyTorch is a framework to enable fully automated deep learning (AutoDL) It combines multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) We show that Auto-PyTorch performs better than several state-of-the-art competitors on average.
• Score: 53.40030379661183
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data. To thoroughly study our assumptions on how to design such an AutoDL system, we additionally introduce a new benchmark on learning curves for DNNs, dubbed LCBench, and run extensive ablation studies of the full Auto-PyTorch on typical AutoML benchmarks, eventually showing that Auto-PyTorch performs better than several state-of-the-art competitors on average.

Related papers

• SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning [14.702694298483445]
  Tree-Search Enhanced LLM Agents (SELA) is an agent-based system that leverages Monte Carlo Tree Search (MCTS) to optimize the AutoML process. SELA represents pipeline configurations as trees, enabling agents to conduct experiments intelligently and iteratively refine their strategies. In an extensive evaluation across 20 machine learning datasets, we compare the performance of traditional and agent-based AutoML methods.
  arXiv  Detail & Related papers  (2024-10-22T17:56:08Z)
• Principled Architecture-aware Scaling of Hyperparameters [69.98414153320894]
  Training a high-quality deep neural network requires choosing suitable hyperparameters, which is a non-trivial and expensive process. In this work, we precisely characterize the dependence of initializations and maximal learning rates on the network architecture. We demonstrate that network rankings can be easily changed by better training networks in benchmarks.
  arXiv  Detail & Related papers  (2024-02-27T11:52:49Z)
• A Deep Neural Networks ensemble workflow from hyperparameter search to inference leveraging GPU clusters [0.0]
  AutoML seeks to automatically build ensembles of Deep Neural Networks (DNNs) to achieve qualitative predictions. We propose a new AutoML to build a larger library of accurate and diverse individual models to then construct ensembles. New ensemble selection method based on a multi-objective greedy algorithm is proposed to generate accurate ensembles.
  arXiv  Detail & Related papers  (2022-08-30T08:04:19Z)
• Efficient Automated Deep Learning for Time Series Forecasting [42.47842694670572]
  We propose an efficient approach for the joint optimization of neural architecture and hyperparameters of the entire data processing pipeline for time series forecasting. In contrast to common NAS search spaces, we designed a novel neural architecture search space covering various state-of-the-art architectures. We empirically study several different budget types enabling efficient multi-fidelity optimization on different forecasting datasets.
  arXiv  Detail & Related papers  (2022-05-11T14:03:25Z)
• AutoBERT-Zero: Evolving BERT Backbone from Scratch [94.89102524181986]
  We propose an Operation-Priority Neural Architecture Search (OP-NAS) algorithm to automatically search for promising hybrid backbone architectures. We optimize both the search algorithm and evaluation of candidate models to boost the efficiency of our proposed OP-NAS. Experiments show that the searched architecture (named AutoBERT-Zero) significantly outperforms BERT and its variants of different model capacities in various downstream tasks.
  arXiv  Detail & Related papers  (2021-07-15T16:46:01Z)
• Resource-Aware Pareto-Optimal Automated Machine Learning Platform [1.6746303554275583]
  novel platform Resource-Aware AutoML (RA-AutoML) RA-AutoML enables flexible and generalized algorithms to build machine learning models subjected to multiple objectives.
  arXiv  Detail & Related papers  (2020-10-30T19:37:48Z)
• Automatic deep learning for trend prediction in time series data [0.0]
  Deep Neural Network (DNN) algorithms have been explored for predicting trends in time series data. In many real world applications, time series data are captured from dynamic systems. We show how a recent AutoML tool can be effectively used to automate the model development process.
  arXiv  Detail & Related papers  (2020-09-17T19:47:05Z)
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation [97.42894942391575]
  We propose FAST-DAD to distill arbitrarily complex ensemble predictors into individual models like boosted trees, random forests, and deep networks. Our individual distilled models are over 10x faster and more accurate than ensemble predictors produced by AutoML tools like H2O/AutoSklearn.
  arXiv  Detail & Related papers  (2020-06-25T09:57:47Z)
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction [75.16836697734995]
  We propose a two-stage algorithm called Automatic Feature Interaction Selection (AutoFIS) AutoFIS can automatically identify important feature interactions for factorization models with computational cost just equivalent to training the target model to convergence. AutoFIS has been deployed onto the training platform of Huawei App Store recommendation service.
  arXiv  Detail & Related papers  (2020-03-25T06:53:54Z)
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data [120.2298620652828]
  We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models. Tests on a suite of 50 classification and regression tasks from Kaggle and the OpenML AutoML Benchmark reveal that AutoGluon is faster, more robust, and much more accurate.
  arXiv  Detail & Related papers  (2020-03-13T23:10:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.