Efficient Multi-Objective Neural Architecture Search via Pareto Dominance-based Novelty Search

• URL: http://arxiv.org/abs/2407.20656v1
• Date: Tue, 30 Jul 2024 08:52:10 GMT
• Title: Efficient Multi-Objective Neural Architecture Search via Pareto Dominance-based Novelty Search
• Authors: An Vo, Ngoc Hoang Luong,
• Abstract summary: Neural Architecture Search (NAS) aims to automate the discovery of high-performing deep neural network architectures. Traditional NAS approaches typically optimize a certain performance metric (e.g., prediction accuracy) overlooking large parts of the architecture search space that potentially contain interesting network configurations. This paper presents a novelty search for multi-objective NAS with Multiple Training-Free metrics (MTF-PDNS)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Architecture Search (NAS) aims to automate the discovery of high-performing deep neural network architectures. Traditional objective-based NAS approaches typically optimize a certain performance metric (e.g., prediction accuracy), overlooking large parts of the architecture search space that potentially contain interesting network configurations. Furthermore, objective-driven population-based metaheuristics in complex search spaces often quickly exhaust population diversity and succumb to premature convergence to local optima. This issue becomes more complicated in NAS when performance objectives do not fully align with the actual performance of the candidate architectures, as is often the case with training-free metrics. While training-free metrics have gained popularity for their rapid performance estimation of candidate architectures without incurring computation-heavy network training, their effective incorporation into NAS remains a challenge. This paper presents the Pareto Dominance-based Novelty Search for multi-objective NAS with Multiple Training-Free metrics (MTF-PDNS). Unlike conventional NAS methods that optimize explicit objectives, MTF-PDNS promotes population diversity by utilizing a novelty score calculated based on multiple training-free performance and complexity metrics, thereby yielding a broader exploration of the search space. Experimental results on standard NAS benchmark suites demonstrate that MTF-PDNS outperforms conventional methods driven by explicit objectives in terms of convergence speed, diversity maintenance, architecture transferability, and computational costs.

Related papers

• A Pairwise Comparison Relation-assisted Multi-objective Evolutionary Neural Architecture Search Method with Multi-population Mechanism [58.855741970337675]
  Neural architecture search (NAS) enables re-searchers to automatically explore vast search spaces and find efficient neural networks. NAS suffers from a key bottleneck, i.e., numerous architectures need to be evaluated during the search process. We propose the SMEM-NAS, a pairwise com-parison relation-assisted multi-objective evolutionary algorithm based on a multi-population mechanism.
  arXiv  Detail & Related papers  (2024-07-22T12:46:22Z)
• Robustifying and Boosting Training-Free Neural Architecture Search [49.828875134088904]
  We propose a robustifying and boosting training-free NAS (RoBoT) algorithm to develop a robust and consistently better-performing metric on diverse tasks. Remarkably, the expected performance of our RoBoT can be theoretically guaranteed, which improves over the existing training-free NAS.
  arXiv  Detail & Related papers  (2024-03-12T12:24:11Z)
• Generalizable Lightweight Proxy for Robust NAS against Diverse Perturbations [59.683234126055694]
  Recent neural architecture search (NAS) frameworks have been successful in finding optimal architectures for given conditions. We propose a novel lightweight robust zero-cost proxy that considers the consistency across features, parameters, and gradients of both clean and perturbed images. Our approach facilitates an efficient and rapid search for neural architectures capable of learning generalizable features that exhibit robustness across diverse perturbations.
  arXiv  Detail & Related papers  (2023-06-08T08:34:26Z)
• Training-free Neural Architecture Search for RNNs and Transformers [0.0]
  We develop a new training-free metric, named hidden covariance, that predicts the trained performance of an RNN architecture. We find that the current search space paradigm for transformer architectures is not optimized for training-free neural architecture search.
  arXiv  Detail & Related papers  (2023-06-01T02:06:13Z)
• You Only Search Once: On Lightweight Differentiable Architecture Search for Resource-Constrained Embedded Platforms [10.11289927237036]
  Differentiable neural architecture search (NAS) has evolved as the most dominant alternative to automatically design competitive deep neural networks (DNNs) We introduce a lightweight hardware-aware differentiable NAS framework dubbed LightNAS, striving to find the required architecture through a one-time search. Extensive experiments are conducted to show the superiority of LightNAS over previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-08-30T02:23:23Z)
• Surrogate-assisted Multi-objective Neural Architecture Search for Real-time Semantic Segmentation [11.866947846619064]
  neural architecture search (NAS) has emerged as a promising avenue toward automating the design of architectures. We propose a surrogate-assisted multi-objective method to address the challenges of applying NAS to semantic segmentation. Our method can identify architectures significantly outperforming existing state-of-the-art architectures designed both manually by human experts and automatically by other NAS methods.
  arXiv  Detail & Related papers  (2022-08-14T10:18:51Z)
• Elastic Architecture Search for Diverse Tasks with Different Resources [87.23061200971912]
  We study a new challenging problem of efficient deployment for diverse tasks with different resources, where the resource constraint and task of interest corresponding to a group of classes are dynamically specified at testing time. Previous NAS approaches seek to design architectures for all classes simultaneously, which may not be optimal for some individual tasks. We present a novel and general framework, called Elastic Architecture Search (EAS), permitting instant specializations at runtime for diverse tasks with various resource constraints.
  arXiv  Detail & Related papers  (2021-08-03T00:54:27Z)
• Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization [29.80410614305851]
  Neural architecture search (NAS) and hyper parameter optimization (HPO) make deep learning accessible to non-experts. We propose a set of methods that extend current approaches to jointly optimize neural architectures and hyper parameters with respect to multiple objectives. These methods will serve as simple baselines for future research on multi-objective joint NAS + HPO.
  arXiv  Detail & Related papers  (2021-05-03T17:04:56Z)
• Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search [70.57382341642418]
  Weight sharing has become a de facto standard in neural architecture search because it enables the search to be done on commodity hardware. Recent works have empirically shown a ranking disorder between the performance of stand-alone architectures and that of the corresponding shared-weight networks. We propose a regularization term that aims to maximize the correlation between the performance rankings of the shared-weight network and that of the standalone architectures.
  arXiv  Detail & Related papers  (2021-04-12T09:32:33Z)
• DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruning [135.27931587381596]
  We propose an efficient and unified NAS framework termed DDPNAS via dynamic distribution pruning. In particular, we first sample architectures from a joint categorical distribution. Then the search space is dynamically pruned and its distribution is updated every few epochs. With the proposed efficient network generation method, we directly obtain the optimal neural architectures on given constraints.
  arXiv  Detail & Related papers  (2019-05-28T06:35:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.