RARTS: An Efficient First-Order Relaxed Architecture Search Method

• URL: http://arxiv.org/abs/2008.03901v2
• Date: Fri, 24 Jun 2022 06:36:21 GMT
• Title: RARTS: An Efficient First-Order Relaxed Architecture Search Method
• Authors: Fanghui Xue, Yingyong Qi, Jack Xin
• Abstract summary: Differentiable architecture search (DARTS) is an effective method for data-driven neural network design based on solving a bilevel optimization problem. We formulate a single level alternative and a relaxed architecture search (RARTS) method that utilizes the whole dataset in architecture learning via both data and network splitting. For the task of searching topological architecture, i.e., the edges and the operations, RARTS obtains a higher accuracy and 60% reduction of computational cost than second-order DARTS on CIFAR-10.
• Score: 5.491655566898372
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentiable architecture search (DARTS) is an effective method for data-driven neural network design based on solving a bilevel optimization problem. Despite its success in many architecture search tasks, there are still some concerns about the accuracy of first-order DARTS and the efficiency of the second-order DARTS. In this paper, we formulate a single level alternative and a relaxed architecture search (RARTS) method that utilizes the whole dataset in architecture learning via both data and network splitting, without involving mixed second derivatives of the corresponding loss functions like DARTS. In our formulation of network splitting, two networks with different but related weights cooperate in search of a shared architecture. The advantage of RARTS over DARTS is justified by a convergence theorem and an analytically solvable model. Moreover, RARTS outperforms DARTS and its variants in accuracy and search efficiency, as shown in adequate experimental results. For the task of searching topological architecture, i.e., the edges and the operations, RARTS obtains a higher accuracy and 60\% reduction of computational cost than second-order DARTS on CIFAR-10. RARTS continues to out-perform DARTS upon transfer to ImageNet and is on par with recent variants of DARTS even though our innovation is purely on the training algorithm without modifying search space. For the task of searching width, i.e., the number of channels in convolutional layers, RARTS also outperforms the traditional network pruning benchmarks. Further experiments on the public architecture search benchmark like NATS-Bench also support the preeminence of RARTS.

Related papers

• OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strength [70.76342136866413]
  Differentiable architecture search (DARTS) has emerged as a promising technique for effective neural architecture search. DARTS suffers from the well-known degeneration issue which can lead to deteriorating architectures. We propose a novel criterion based on operation strength that estimates the importance of an operation by its effect on the final loss.
  arXiv  Detail & Related papers  (2024-09-22T13:16:07Z)
• Efficient Architecture Search via Bi-level Data Pruning [70.29970746807882]
  This work pioneers an exploration into the critical role of dataset characteristics for DARTS bi-level optimization. We introduce a new progressive data pruning strategy that utilizes supernet prediction dynamics as the metric. Comprehensive evaluations on the NAS-Bench-201 search space, DARTS search space, and MobileNet-like search space validate that BDP reduces search costs by over 50%.
  arXiv  Detail & Related papers  (2023-12-21T02:48:44Z)
• $\Lambda$-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells [11.777101481512423]
  Differentiable neural architecture search (DARTS) is a popular method for neural architecture search (NAS) We show that DARTS suffers from a specific structural flaw due to its weight-sharing framework that limits the convergence of DARTS to saturation points of the softmax function. We propose two new regularization terms that aim to prevent performance collapse by harmonizing operation selection via aligning gradients of layers.
  arXiv  Detail & Related papers  (2022-10-14T17:54:01Z)
• D-DARTS: Distributed Differentiable Architecture Search [75.12821786565318]
  Differentiable ARchiTecture Search (DARTS) is one of the most trending Neural Architecture Search (NAS) methods. We propose D-DARTS, a novel solution that addresses this problem by nesting several neural networks at cell-level.
  arXiv  Detail & Related papers  (2021-08-20T09:07:01Z)
• DARTS for Inverse Problems: a Study on Hyperparameter Sensitivity [21.263326724329698]
  Differentiable architecture search (DARTS) is a widely researched tool for neural architecture search. We report the results of any DARTS-based methods from several runs along with its underlying performance statistics.
  arXiv  Detail & Related papers  (2021-08-12T10:28:02Z)
• iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients [75.41173109807735]
  Differentiable ARchiTecture Search (DARTS) has recently become the mainstream of neural architecture search (NAS) We tackle the hypergradient computation in DARTS based on the implicit function theorem. We show that the architecture optimisation with the proposed method, named iDARTS, is expected to converge to a stationary point.
  arXiv  Detail & Related papers  (2021-06-21T00:44:11Z)
• Partially-Connected Differentiable Architecture Search for Deepfake and Spoofing Detection [14.792884010821762]
  This paper reports the first successful application of a differentiable architecture search (DARTS) approach to the deepfake and spoofing detection problems. DARTS operates upon a continuous, differentiable search space which enables both the architecture and parameters to be optimised via gradient descent.
  arXiv  Detail & Related papers  (2021-04-07T13:53:20Z)
• Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search [50.40004966087121]
  We introduce a new reinforcement learning based neural architecture search (NAS) methodology for generative adversarial network (GAN) architecture search. The key idea is to formulate the GAN architecture search problem as a Markov decision process (MDP) for smoother architecture sampling. We exploit an off-policy GAN architecture search algorithm that makes efficient use of the samples generated by previous policies.
  arXiv  Detail & Related papers  (2020-07-17T18:29:17Z)
• Theory-Inspired Path-Regularized Differential Network Architecture Search [206.93821077400733]
  We study the impact of skip connections to fast network optimization and its competitive advantage over other types of operations in differential architecture search (DARTS) We propose a theory-inspired path-regularized DARTS that consists of two key modules: (i) a differential group-structured sparse binary gate introduced for each operation to avoid unfair competition among operations, and (ii) a path-depth-wise regularization used to incite search exploration for deep architectures that converge slower than shallow ones.
  arXiv  Detail & Related papers  (2020-06-30T05:28:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.