ModuleNet: Knowledge-inherited Neural Architecture Search

• URL: http://arxiv.org/abs/2004.05020v2
• Date: Tue, 14 Apr 2020 03:39:26 GMT
• Title: ModuleNet: Knowledge-inherited Neural Architecture Search
• Authors: Yaran Chen, Ruiyuan Gao, Fenggang Liu and Dongbin Zhao
• Abstract summary: We discuss what kind of knowledge in a model can and should be used for new architecture design. We propose a new NAS algorithm, namely ModuleNet, which can fully inherit knowledge from existing convolutional neural networks. Our strategy can efficiently evaluate the performance of new architecture even without tuning weights in convolutional layers.
• Score: 7.769061374951596
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although Neural Architecture Search (NAS) can bring improvement to deep models, they always neglect precious knowledge of existing models. The computation and time costing property in NAS also means that we should not start from scratch to search, but make every attempt to reuse the existing knowledge. In this paper, we discuss what kind of knowledge in a model can and should be used for new architecture design. Then, we propose a new NAS algorithm, namely ModuleNet, which can fully inherit knowledge from existing convolutional neural networks. To make full use of existing models, we decompose existing models into different \textit{module}s which also keep their weights, consisting of a knowledge base. Then we sample and search for new architecture according to the knowledge base. Unlike previous search algorithms, and benefiting from inherited knowledge, our method is able to directly search for architectures in the macro space by NSGA-II algorithm without tuning parameters in these \textit{module}s. Experiments show that our strategy can efficiently evaluate the performance of new architecture even without tuning weights in convolutional layers. With the help of knowledge we inherited, our search results can always achieve better performance on various datasets (CIFAR10, CIFAR100) over original architectures.

Related papers

• Knowledge-aware Evolutionary Graph Neural Architecture Search [49.13787973318586]
  Graph neural architecture search (GNAS) can customize high-performance graph neural network architectures for specific graph tasks or datasets. Existing GNAS methods begin searching for architectures from a zero-knowledge state, ignoring the prior knowledge that may improve the search efficiency. This study proposes exploiting such prior knowledge to accelerate the multi-objective evolutionary search on a new graph dataset.
  arXiv  Detail & Related papers  (2024-11-26T11:32:45Z)
• Building Optimal Neural Architectures using Interpretable Knowledge [15.66288233048004]
  AutoBuild is a scheme which learns to align the latent embeddings of operations and architecture modules with the ground-truth performance of the architectures they appear in. We show that by mining a relatively small set of evaluated architectures, AutoBuild can learn to build high-quality architectures directly or help to reduce search space to focus on relevant areas.
  arXiv  Detail & Related papers  (2024-03-20T04:18:38Z)
• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• GeNAS: Neural Architecture Search with Better Generalization [14.92869716323226]
  Recent neural architecture search (NAS) approaches rely on validation loss or accuracy to find the superior network for the target data. In this paper, we investigate a new neural architecture search measure for excavating architectures with better generalization.
  arXiv  Detail & Related papers  (2023-05-15T12:44:54Z)
• NASiam: Efficient Representation Learning using Neural Architecture Search for Siamese Networks [76.8112416450677]
  Siamese networks are one of the most trending methods to achieve self-supervised visual representation learning (SSL) NASiam is a novel approach that uses for the first time differentiable NAS to improve the multilayer perceptron projector and predictor (encoder/predictor pair) NASiam reaches competitive performance in both small-scale (i.e., CIFAR-10/CIFAR-100) and large-scale (i.e., ImageNet) image classification datasets while costing only a few GPU hours.
  arXiv  Detail & Related papers  (2023-01-31T19:48:37Z)
• Automating Neural Architecture Design without Search [3.651848964235307]
  We study the automated architecture design from a new perspective that eliminates the need to sequentially evaluate each neural architecture generated during algorithm execution. We implement the proposed approach by using a graph neural network for link prediction and acquired the knowledge from NAS-Bench-101. In addition, we also utilized the learned knowledge from NAS-Bench-101 to automate architecture design in the DARTS search space, and achieved 97.82% accuracy on CIFAR10, and 76.51% top-1 accuracy on ImageNet consuming only $2times10-4$ GPU days.
  arXiv  Detail & Related papers  (2022-04-21T14:41:05Z)
• Network Graph Based Neural Architecture Search [57.78724765340237]
  We search neural network by rewiring the corresponding graph and predict the architecture performance by graph properties. Because we do not perform machine learning over the entire graph space, the searching process is remarkably efficient.
  arXiv  Detail & Related papers  (2021-12-15T00:12:03Z)
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule [95.56873042777316]
  Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost. This paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions. We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability.
  arXiv  Detail & Related papers  (2021-11-25T18:13:42Z)
• Contrastive Neural Architecture Search with Neural Architecture Comparators [46.45102111497492]
  One of the key steps in Neural Architecture Search (NAS) is to estimate the performance of candidate architectures. Existing methods either directly use the validation performance or learn a predictor to estimate the performance. We propose a novel Contrastive Neural Architecture Search (CTNAS) method which performs architecture search by taking the comparison results between architectures as the reward.
  arXiv  Detail & Related papers  (2021-03-08T11:24:07Z)
• Learning Architectures from an Extended Search Space for Language Modeling [37.79977691127229]
  We present a general approach to learn both intra-cell and inter-cell architectures of Neural architecture search (NAS) For recurrent neural language modeling, it outperforms a strong baseline significantly on the PTB and WikiText data, with a new state-of-the-art on PTB. The learned architectures show good transferability to other systems.
  arXiv  Detail & Related papers  (2020-05-06T05:02:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.