Single-DARTS: Towards Stable Architecture Search

• URL: http://arxiv.org/abs/2108.08128v1
• Date: Wed, 18 Aug 2021 13:00:39 GMT
• Title: Single-DARTS: Towards Stable Architecture Search
• Authors: Pengfei Hou, Ying Jin, Yukang Chen
• Abstract summary: We propose Single-DARTS, which merely uses single-level optimization, updating network weights and architecture parameters simultaneously with the same data batch. Experiment results show that Single-DARTS achieves state-of-the-art performance on mainstream search spaces.
• Score: 7.894638544388165
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentiable architecture search (DARTS) marks a milestone in Neural Architecture Search (NAS), boasting simplicity and small search costs. However, DARTS still suffers from frequent performance collapse, which happens when some operations, such as skip connections, zeroes and poolings, dominate the architecture. In this paper, we are the first to point out that the phenomenon is attributed to bi-level optimization. We propose Single-DARTS which merely uses single-level optimization, updating network weights and architecture parameters simultaneously with the same data batch. Even single-level optimization has been previously attempted, no literature provides a systematic explanation on this essential point. Replacing the bi-level optimization, Single-DARTS obviously alleviates performance collapse as well as enhances the stability of architecture search. Experiment results show that Single-DARTS achieves state-of-the-art performance on mainstream search spaces. For instance, on NAS-Benchmark-201, the searched architectures are nearly optimal ones. We also validate that the single-level optimization framework is much more stable than the bi-level one. We hope that this simple yet effective method will give some insights on differential architecture search. The code is available at https://github.com/PencilAndBike/Single-DARTS.git.

Related papers

• Mutually-aware Sub-Graphs Differentiable Architecture Search [29.217547815683748]
  Mutually-aware Sub-Graphs Differentiable Architecture Search (MSG-DAS) MSG-DAS is a differentiable Gumbel-TopK sampler that produces multiple mutually exclusive single-path sub-graphs. We demonstrate the effectiveness of our methods on ImageNet and CIFAR10, where the searched models show a comparable performance as the most recent approaches.
  arXiv  Detail & Related papers  (2021-07-09T09:31:31Z)
• iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients [75.41173109807735]
  Differentiable ARchiTecture Search (DARTS) has recently become the mainstream of neural architecture search (NAS) We tackle the hypergradient computation in DARTS based on the implicit function theorem. We show that the architecture optimisation with the proposed method, named iDARTS, is expected to converge to a stationary point.
  arXiv  Detail & Related papers  (2021-06-21T00:44:11Z)
• Making Differentiable Architecture Search less local [9.869449181400466]
  Differentiable neural architecture search (DARTS) is a promising NAS approach that dramatically increases search efficiency. It has been shown to suffer from performance collapse, where the search often leads to detrimental architectures. We develop a more global optimisation scheme that is able to better explore the space without changing the DARTS problem formulation.
  arXiv  Detail & Related papers  (2021-04-21T10:36:43Z)
• Weak NAS Predictors Are All You Need [91.11570424233709]
  Recent predictor-based NAS approaches attempt to solve the problem with two key steps: sampling some architecture-performance pairs and fitting a proxy accuracy predictor. We shift the paradigm from finding a complicated predictor that covers the whole architecture space to a set of weaker predictors that progressively move towards the high-performance sub-space. Our method costs fewer samples to find the top-performance architectures on NAS-Bench-101 and NAS-Bench-201, and it achieves the state-of-the-art ImageNet performance on the NASNet search space.
  arXiv  Detail & Related papers  (2021-02-21T01:58:43Z)
• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search [84.4140192638394]
  Most differentiable neural architecture search methods construct a super-net for search and derive a target-net as its sub-graph for evaluation. In this paper, we introduce EnTranNAS that is composed of Engine-cells and Transit-cells. Our method also spares much memory and computation cost, which speeds up the search process.
  arXiv  Detail & Related papers  (2021-01-27T12:16:47Z)
• Single-level Optimization For Differential Architecture Search [6.3531384587183135]
  differential architecture search (DARTS) makes gradient of architecture parameters biased for network weights. We propose to use single-level to replace bi-level optimization and non-competitive activation function like sigmoid to replace softmax. Experiments on NAS Benchmark 201 validate our hypothesis and stably find out nearly the optimal architecture.
  arXiv  Detail & Related papers  (2020-12-15T18:40:33Z)
• ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradient Accumulation [106.04777600352743]
  Differentiable architecture search (DARTS) is largely hindered by its substantial memory cost since the entire supernet resides in the memory. The single-path DARTS comes in, which only chooses a single-path submodel at each step. While being memory-friendly, it also comes with low computational costs. We propose a new algorithm called RObustifying Memory-Efficient NAS (ROME) to give a cure.
  arXiv  Detail & Related papers  (2020-11-23T06:34:07Z)
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding [86.40042104698792]
  We formulate neural architecture search as a sparse coding problem. In experiments, our two-stage method on CIFAR-10 requires only 0.05 GPU-day for search. Our one-stage method produces state-of-the-art performances on both CIFAR-10 and ImageNet at the cost of only evaluation time.
  arXiv  Detail & Related papers  (2020-10-13T04:34:24Z)
• GOLD-NAS: Gradual, One-Level, Differentiable [100.12492801459105]
  We propose a novel algorithm named Gradual One-Level Differentiable Neural Architecture Search (GOLD-NAS) It introduces a variable resource constraint to one-level optimization so that the weak operators are gradually pruned out from the super-network.
  arXiv  Detail & Related papers  (2020-07-07T10:37:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.