DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture

• URL: http://arxiv.org/abs/2109.05765v1
• Date: Mon, 13 Sep 2021 08:12:50 GMT
• Title: DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Authors: Kaichen Zhou, Lanqing Hong, Shoukang Hu, Fengwei Zhou, Binxin Ru, Jiashi Feng, Zhenguo Li
• Abstract summary: We propose an end-to-end solution that integrates the AutoML components and returns a ready-to-use model at the end of the search. Dha achieves state-of-the-art (SOTA) results on various datasets, especially 77.4% accuracy on ImageNet with cell based search space.
• Score: 81.82173855071312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated machine learning (AutoML) usually involves several crucial components, such as Data Augmentation (DA) policy, Hyper-Parameter Optimization (HPO), and Neural Architecture Search (NAS). Although many strategies have been developed for automating these components in separation, joint optimization of these components remains challenging due to the largely increased search dimension and the variant input types of each component. Meanwhile, conducting these components in a sequence often requires careful coordination by human experts and may lead to sub-optimal results. In parallel to this, the common practice of searching for the optimal architecture first and then retraining it before deployment in NAS often suffers from low performance correlation between the search and retraining stages. An end-to-end solution that integrates the AutoML components and returns a ready-to-use model at the end of the search is desirable. In view of these, we propose DHA, which achieves joint optimization of Data augmentation policy, Hyper-parameter and Architecture. Specifically, end-to-end NAS is achieved in a differentiable manner by optimizing a compressed lower-dimensional feature space, while DA policy and HPO are updated dynamically at the same time. Experiments show that DHA achieves state-of-the-art (SOTA) results on various datasets, especially 77.4\% accuracy on ImageNet with cell based search space, which is higher than current SOTA by 0.5\%. To the best of our knowledge, we are the first to efficiently and jointly optimize DA policy, NAS, and HPO in an end-to-end manner without retraining.

Related papers

• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. We increase the consistency and informativeness of the pairwise preference signals through targeted modifications. We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Efficient Architecture Search via Bi-level Data Pruning [70.29970746807882]
  This work pioneers an exploration into the critical role of dataset characteristics for DARTS bi-level optimization. We introduce a new progressive data pruning strategy that utilizes supernet prediction dynamics as the metric. Comprehensive evaluations on the NAS-Bench-201 search space, DARTS search space, and MobileNet-like search space validate that BDP reduces search costs by over 50%.
  arXiv  Detail & Related papers  (2023-12-21T02:48:44Z)
• Fairer and More Accurate Tabular Models Through NAS [14.147928131445852]
  We propose using multi-objective Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) in the first application to the very challenging domain of tabular data. We show that models optimized solely for accuracy with NAS often fail to inherently address fairness concerns. We produce architectures that consistently dominate state-of-the-art bias mitigation methods either in fairness, accuracy or both.
  arXiv  Detail & Related papers  (2023-10-18T17:56:24Z)
• HomOpt: A Homotopy-Based Hyperparameter Optimization Method [10.11271414863925]
  We propose HomOpt, a data-driven approach based on a generalized additive model (GAM) surrogate combined with homotopy optimization. We show how HomOpt can boost the performance and effectiveness of any given method with faster convergence to the optimum on continuous discrete, and categorical domain spaces.
  arXiv  Detail & Related papers  (2023-08-07T06:01:50Z)
• Efficient Automated Deep Learning for Time Series Forecasting [42.47842694670572]
  We propose an efficient approach for the joint optimization of neural architecture and hyperparameters of the entire data processing pipeline for time series forecasting. In contrast to common NAS search spaces, we designed a novel neural architecture search space covering various state-of-the-art architectures. We empirically study several different budget types enabling efficient multi-fidelity optimization on different forecasting datasets.
  arXiv  Detail & Related papers  (2022-05-11T14:03:25Z)
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning [72.54359545547904]
  We propose a gradient-based subset selection framework for hyper- parameter tuning. We show that using gradient-based data subsets for hyper- parameter tuning achieves significantly faster turnaround times and speedups of 3$times$-30$times$.
  arXiv  Detail & Related papers  (2022-03-15T19:25:01Z)
• DAAS: Differentiable Architecture and Augmentation Policy Search [107.53318939844422]
  This work considers the possible coupling between neural architectures and data augmentation and proposes an effective algorithm jointly searching for them. Our approach achieves 97.91% accuracy on CIFAR-10 and 76.6% Top-1 accuracy on ImageNet dataset, showing the outstanding performance of our search algorithm.
  arXiv  Detail & Related papers  (2021-09-30T17:15:17Z)
• Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization [29.80410614305851]
  Neural architecture search (NAS) and hyper parameter optimization (HPO) make deep learning accessible to non-experts. We propose a set of methods that extend current approaches to jointly optimize neural architectures and hyper parameters with respect to multiple objectives. These methods will serve as simple baselines for future research on multi-objective joint NAS + HPO.
  arXiv  Detail & Related papers  (2021-05-03T17:04:56Z)
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery [54.60327265077322]
  We study two important problems in the automated design of neural networks through the lens of sparse recovery methods. In the first part of this paper, we establish a novel connection between HPO and structured sparse recovery. In the second part of this paper, we establish a connection between NAS and structured sparse recovery.
  arXiv  Detail & Related papers  (2020-07-07T00:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.