RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving

• URL: http://arxiv.org/abs/2108.08019v1
• Date: Wed, 18 Aug 2021 07:45:21 GMT
• Title: RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving
• Authors: Ruochen Wang, Xiangning Chen, Minhao Cheng, Xiaocheng Tang, Cho-Jui Hsieh
• Abstract summary: We propose NOn-uniform Successive Halving (NOSH), a hierarchical scheduling algorithm that terminates the training of underperforming architectures early to avoid wasting budget. We formulate predictor-based architecture search as learning to rank with pairwise comparisons. The resulting method - RANK-NOSH, reduces the search budget by 5x while achieving competitive or even better performance than previous state-of-the-art predictor-based methods on various spaces and datasets.
• Score: 74.61723678821049
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predictor-based algorithms have achieved remarkable performance in the Neural Architecture Search (NAS) tasks. However, these methods suffer from high computation costs, as training the performance predictor usually requires training and evaluating hundreds of architectures from scratch. Previous works along this line mainly focus on reducing the number of architectures required to fit the predictor. In this work, we tackle this challenge from a different perspective - improve search efficiency by cutting down the computation budget of architecture training. We propose NOn-uniform Successive Halving (NOSH), a hierarchical scheduling algorithm that terminates the training of underperforming architectures early to avoid wasting budget. To effectively leverage the non-uniform supervision signals produced by NOSH, we formulate predictor-based architecture search as learning to rank with pairwise comparisons. The resulting method - RANK-NOSH, reduces the search budget by ~5x while achieving competitive or even better performance than previous state-of-the-art predictor-based methods on various spaces and datasets.

Related papers

• Neural Algorithmic Reasoning Without Intermediate Supervision [21.852775399735005]
  We focus on learning neural algorithmic reasoning only from the input-output pairs without appealing to the intermediate supervision. We build a self-supervised objective that can regularise intermediate computations of the model without access to the algorithm trajectory. We demonstrate that our approach is competitive to its trajectory-supervised counterpart on tasks from the CLRSic Algorithmic Reasoning Benchmark.
  arXiv  Detail & Related papers  (2023-06-23T09:57:44Z)
• AIO-P: Expanding Neural Performance Predictors Beyond Image Classification [22.743278613519152]
  We propose a novel All-in-One Predictor (AIO-P) to pretrain neural predictors on architecture examples. AIO-P can achieve Mean Absolute Error (MAE) and Spearman's Rank Correlation (SRCC) below 1% and above 0.5, respectively.
  arXiv  Detail & Related papers  (2022-11-30T18:30:41Z)
• EmProx: Neural Network Performance Estimation For Neural Architecture Search [0.0]
  This study proposes a new method, EmProx Score (Embedding Proximity Score) to map architectures to a continuous embedding space. Performance of candidates is then estimated using weighted kNN based on the embedding vectors of architectures of which the performance is known. Performance estimations of this method are comparable to the performance predictor used in NAO in terms of accuracy, while being nearly nine times faster to train compared to NAO.
  arXiv  Detail & Related papers  (2022-06-13T08:35:52Z)
• Non-Clairvoyant Scheduling with Predictions Revisited [77.86290991564829]
  In non-clairvoyant scheduling, the task is to find an online strategy for scheduling jobs with a priori unknown processing requirements. We revisit this well-studied problem in a recently popular learning-augmented setting that integrates (untrusted) predictions in algorithm design. We show that these predictions have desired properties, admit a natural error measure as well as algorithms with strong performance guarantees.
  arXiv  Detail & Related papers  (2022-02-21T13:18:11Z)
• How Powerful are Performance Predictors in Neural Architecture Search? [43.86743225322636]
  We give the first large-scale study of performance predictors by analyzing 31 techniques. We show that certain families of predictors can be combined to achieve even better predictive power.
  arXiv  Detail & Related papers  (2021-04-02T17:57:16Z)
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets [93.79297053429447]
  Existing methods often perform an independent architecture search for each target budget. We propose a general architecture generator that automatically produces effective architectures for an arbitrary budget merely via model inference. Extensive experiments on three platforms (i.e., mobile, CPU, and GPU) show the superiority of the proposed method over existing NAS methods.
  arXiv  Detail & Related papers  (2021-02-27T13:59:17Z)
• Multi-objective Neural Architecture Search with Almost No Training [9.93048700248444]
  We propose an effective alternative, dubbed Random-Weight Evaluation (RWE), to rapidly estimate the performance of network architectures. RWE reduces the computational cost of evaluating an architecture from hours to seconds. When integrated within an evolutionary multi-objective algorithm, RWE obtains a set of efficient architectures with state-of-the-art performance on CIFAR-10 with less than two hours' searching on a single GPU card.
  arXiv  Detail & Related papers  (2020-11-27T07:39:17Z)
• Evolving Search Space for Neural Architecture Search [70.71153433676024]
  We present a Neural Search-space Evolution (NSE) scheme that amplifies the results from the previous effort by maintaining an optimized search space subset. We achieve 77.3% top-1 retrain accuracy on ImageNet with 333M FLOPs, which yielded a state-of-the-art performance. When the latency constraint is adopted, our result also performs better than the previous best-performing mobile models with a 77.9% Top-1 retrain accuracy.
  arXiv  Detail & Related papers  (2020-11-22T01:11:19Z)
• Speedy Performance Estimation for Neural Architecture Search [47.683124540824515]
  We propose to estimate the final test performance based on a simple measure of training speed. Our estimator is theoretically motivated by the connection between generalisation and training speed.
  arXiv  Detail & Related papers  (2020-06-08T11:48:09Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.