Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

• URL: http://arxiv.org/abs/2101.11342v1
• Date: Wed, 27 Jan 2021 12:16:47 GMT
• Title: Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search
• Authors: Yibo Yang, Shan You, Hongyang Li, Fei Wang, Chen Qian, Zhouchen Lin
• Abstract summary: Most differentiable neural architecture search methods construct a super-net for search and derive a target-net as its sub-graph for evaluation. In this paper, we introduce EnTranNAS that is composed of Engine-cells and Transit-cells. Our method also spares much memory and computation cost, which speeds up the search process.
• Score: 84.4140192638394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most differentiable neural architecture search methods construct a super-net for search and derive a target-net as its sub-graph for evaluation. There exists a significant gap between the architectures in search and evaluation. As a result, current methods suffer from an inconsistent, inefficient, and inflexible search process. In this paper, we introduce EnTranNAS that is composed of Engine-cells and Transit-cells. The Engine-cell is differentiable for architecture search, while the Transit-cell only transits a sub-graph by architecture derivation. Consequently, the gap between the architectures in search and evaluation is significantly reduced. Our method also spares much memory and computation cost, which speeds up the search process. A feature sharing strategy is introduced for more balanced optimization and more efficient search. Furthermore, we develop an architecture derivation method to replace the traditional one that is based on a hand-crafted rule. Our method enables differentiable sparsification, and keeps the derived architecture equivalent to that of Engine-cell, which further improves the consistency between search and evaluation. Besides, it supports the search for topology where a node can be connected to prior nodes with any number of connections, so that the searched architectures could be more flexible. For experiments on CIFAR-10, our search on the standard space requires only 0.06 GPU-day. We further have an error rate of 2.22% with 0.07 GPU-day for the search on an extended space. We can also directly perform the search on ImageNet with topology learnable and achieve a top-1 error rate of 23.8% in 2.1 GPU-day.

Related papers

• DASS: Differentiable Architecture Search for Sparse neural networks [0.5735035463793009]
  We find that the architectures designed for dense networks by differentiable architecture search methods are ineffective when pruning mechanisms are applied to them. In this paper, we propose a new method to search for sparsity-friendly neural architectures. We do this by adding two new sparse operations to the search space and modifying the search objective.
  arXiv  Detail & Related papers  (2022-07-14T14:53:50Z)
• Partially-Connected Differentiable Architecture Search for Deepfake and Spoofing Detection [14.792884010821762]
  This paper reports the first successful application of a differentiable architecture search (DARTS) approach to the deepfake and spoofing detection problems. DARTS operates upon a continuous, differentiable search space which enables both the architecture and parameters to be optimised via gradient descent.
  arXiv  Detail & Related papers  (2021-04-07T13:53:20Z)
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding [86.40042104698792]
  We formulate neural architecture search as a sparse coding problem. In experiments, our two-stage method on CIFAR-10 requires only 0.05 GPU-day for search. Our one-stage method produces state-of-the-art performances on both CIFAR-10 and ImageNet at the cost of only evaluation time.
  arXiv  Detail & Related papers  (2020-10-13T04:34:24Z)
• Disentangled Neural Architecture Search [7.228790381070109]
  We propose disentangled neural architecture search (DNAS) which disentangles the hidden representation of the controller into semantically meaningful concepts. DNAS successfully disentangles the architecture representations, including operation selection, skip connections, and number of layers. Dense-sampling leads to neural architecture search with higher efficiency and better performance.
  arXiv  Detail & Related papers  (2020-09-24T03:35:41Z)
• Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search [50.40004966087121]
  We introduce a new reinforcement learning based neural architecture search (NAS) methodology for generative adversarial network (GAN) architecture search. The key idea is to formulate the GAN architecture search problem as a Markov decision process (MDP) for smoother architecture sampling. We exploit an off-policy GAN architecture search algorithm that makes efficient use of the samples generated by previous policies.
  arXiv  Detail & Related papers  (2020-07-17T18:29:17Z)
• GOLD-NAS: Gradual, One-Level, Differentiable [100.12492801459105]
  We propose a novel algorithm named Gradual One-Level Differentiable Neural Architecture Search (GOLD-NAS) It introduces a variable resource constraint to one-level optimization so that the weak operators are gradually pruned out from the super-network.
  arXiv  Detail & Related papers  (2020-07-07T10:37:49Z)
• Multi-Objective Neural Architecture Search Based on Diverse Structures and Adaptive Recommendation [4.595675084986132]
  The search space of neural architecture search (NAS) for convolutional neural network (CNN) is huge. We propose MoARR algorithm, which utilizes the existing research results and historical information to quickly find architectures that are both lightweight and accurate. Experimental results show that our MoARR can achieve a powerful and lightweight model (with 1.9% error rate and 2.3M parameters) on CIFAR-10 in 6 GPU hours.
  arXiv  Detail & Related papers  (2020-07-06T13:42:33Z)
• Cyclic Differentiable Architecture Search [99.12381460261841]
  Differentiable ARchiTecture Search, i.e., DARTS, has drawn great attention in neural architecture search. We propose new joint objectives and a novel Cyclic Differentiable ARchiTecture Search framework, dubbed CDARTS. In the DARTS search space, we achieve 97.52% top-1 accuracy on CIFAR10 and 76.3% top-1 accuracy on ImageNet.
  arXiv  Detail & Related papers  (2020-06-18T17:55:19Z)
• Latency-Aware Differentiable Neural Architecture Search [113.35689580508343]
  Differentiable neural architecture search methods became popular in recent years, mainly due to their low search costs and flexibility in designing the search space. However, these methods suffer the difficulty in optimizing network, so that the searched network is often unfriendly to hardware. This paper deals with this problem by adding a differentiable latency loss term into optimization, so that the search process can tradeoff between accuracy and latency with a balancing coefficient.
  arXiv  Detail & Related papers  (2020-01-17T15:55:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.