Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets

• URL: http://arxiv.org/abs/2103.00219v1
• Date: Sat, 27 Feb 2021 13:59:17 GMT
• Title: Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets
• Authors: Yong Guo, Yaofo Chen, Yin Zheng, Qi Chen, Peilin Zhao, Jian Chen, Junzhou Huang, Mingkui Tan
• Abstract summary: Existing methods often perform an independent architecture search for each target budget. We propose a general architecture generator that automatically produces effective architectures for an arbitrary budget merely via model inference. Extensive experiments on three platforms (i.e., mobile, CPU, and GPU) show the superiority of the proposed method over existing NAS methods.
• Score: 93.79297053429447
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Designing feasible and effective architectures under diverse computation budgets incurred by different applications/devices is essential for deploying deep models in practice. Existing methods often perform an independent architecture search for each target budget, which is very inefficient yet unnecessary. Moreover, the repeated independent search manner would inevitably ignore the common knowledge among different search processes and hamper the search performance. To address these issues, we seek to train a general architecture generator that automatically produces effective architectures for an arbitrary budget merely via model inference. To this end, we propose a Pareto-Frontier-aware Neural Architecture Generator (NAG) which takes an arbitrary budget as input and produces the Pareto optimal architecture for the target budget. We train NAG by learning the Pareto frontier (i.e., the set of Pareto optimal architectures) over model performance and computational cost (e.g., latency). Extensive experiments on three platforms (i.e., mobile, CPU, and GPU) show the superiority of the proposed method over existing NAS methods.

Related papers

• Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversity [12.52012450501367]
  Hardware-aware Neural Architecture Search approaches (HW-NAS) automate the design of deep learning architectures tailored specifically to a given target hardware platform. These techniques demand substantial computational resources, primarily due to the expensive process of assessing the performance of identified architectures. We propose a Multi-Objective method to address the HW-NAS problem, called MO-HDNAS, to identify the trade-off set of architectures in a single run with low computational cost.
  arXiv  Detail & Related papers  (2024-04-15T15:32:58Z)
• Multi-objective Differentiable Neural Architecture Search [58.67218773054753]
  We propose a novel NAS algorithm that encodes user preferences for the trade-off between performance and hardware metrics. Our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets.
  arXiv  Detail & Related papers  (2024-02-28T10:09:04Z)
• POPNASv3: a Pareto-Optimal Neural Architecture Search Solution for Image and Time Series Classification [8.190723030003804]
  This article presents the third version of a sequential model-based NAS algorithm targeting different hardware environments and multiple classification tasks. Our method is able to find competitive architectures within large search spaces, while keeping a flexible structure and data processing pipeline to adapt to different tasks. The experiments performed on images and time series classification datasets provide evidence that POPNASv3 can explore a large set of assorted operators and converge to optimal architectures suited for the type of data provided under different scenarios.
  arXiv  Detail & Related papers  (2022-12-13T17:14:14Z)
• Pareto-aware Neural Architecture Generation for Diverse Computational Budgets [94.27982238384847]
  Existing methods often perform an independent architecture search process for each target budget. We propose a Neural Architecture Generator (PNAG) which only needs to be trained once and dynamically produces the optimal architecture for any given budget via inference. Such a joint search algorithm not only greatly reduces the overall search cost but also improves the results.
  arXiv  Detail & Related papers  (2022-10-14T08:30:59Z)
• FocusFormer: Focusing on What We Need via Architecture Sampler [45.150346855368]
  Vision Transformers (ViTs) have underpinned the recent breakthroughs in computer vision. One-shot neural architecture search decouples the supernet training and architecture specialization for diverse deployment scenarios. We devise a simple yet effective method, called FocusFormer, to bridge such a gap.
  arXiv  Detail & Related papers  (2022-08-23T10:42:56Z)
• Efficient Search of Multiple Neural Architectures with Different Complexities via Importance Sampling [3.759936323189417]
  This study focuses on the architecture complexity-aware one-shot NAS that optimize the objective function composed of the weighted sum of two metrics. The proposed method is applied to the architecture search of convolutional neural networks on the CIAFR-10 and ImageNet datasets.
  arXiv  Detail & Related papers  (2022-07-21T07:06:03Z)
• RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving [74.61723678821049]
  We propose NOn-uniform Successive Halving (NOSH), a hierarchical scheduling algorithm that terminates the training of underperforming architectures early to avoid wasting budget. We formulate predictor-based architecture search as learning to rank with pairwise comparisons. The resulting method - RANK-NOSH, reduces the search budget by 5x while achieving competitive or even better performance than previous state-of-the-art predictor-based methods on various spaces and datasets.
  arXiv  Detail & Related papers  (2021-08-18T07:45:21Z)
• One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking [97.60915598958968]
  We propose a one-shot neural ensemble architecture search (NEAS) solution that addresses the two challenges. For the first challenge, we introduce a novel diversity-based metric to guide search space shrinking. For the second challenge, we enable a new search dimension to learn layer sharing among different models for efficiency purposes.
  arXiv  Detail & Related papers  (2021-04-01T16:29:49Z)
• Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search [50.40004966087121]
  We introduce a new reinforcement learning based neural architecture search (NAS) methodology for generative adversarial network (GAN) architecture search. The key idea is to formulate the GAN architecture search problem as a Markov decision process (MDP) for smoother architecture sampling. We exploit an off-policy GAN architecture search algorithm that makes efficient use of the samples generated by previous policies.
  arXiv  Detail & Related papers  (2020-07-17T18:29:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.