$\beta$-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search

• URL: http://arxiv.org/abs/2301.06393v1
• Date: Mon, 16 Jan 2023 12:30:32 GMT
• Title: $\beta$-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search
• Authors: Peng Ye, Tong He, Baopu Li, Tao Chen, Lei Bai, Wanli Ouyang
• Abstract summary: Regularization method, Beta-Decay, is proposed to regularize the DARTS-based NAS searching process (i.e., $beta$-DARTS) In-depth theoretical analyses on how it works and why it works are provided.
• Score: 96.99525100285084
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Architecture Search has attracted increasing attention in recent years. Among them, differential NAS approaches such as DARTS, have gained popularity for the search efficiency. However, they still suffer from three main issues, that are, the weak stability due to the performance collapse, the poor generalization ability of the searched architectures, and the inferior robustness to different kinds of proxies. To solve the stability and generalization problems, a simple-but-effective regularization method, termed as Beta-Decay, is proposed to regularize the DARTS-based NAS searching process (i.e., $\beta$-DARTS). Specifically, Beta-Decay regularization can impose constraints to keep the value and variance of activated architecture parameters from being too large, thereby ensuring fair competition among architecture parameters and making the supernet less sensitive to the impact of input on the operation set. In-depth theoretical analyses on how it works and why it works are provided. Comprehensive experiments validate that Beta-Decay regularization can help to stabilize the searching process and makes the searched network more transferable across different datasets. To address the robustness problem, we first benchmark different NAS methods under a wide range of proxy data, proxy channels, proxy layers and proxy epochs, since the robustness of NAS under different kinds of proxies has not been explored before. We then conclude some interesting findings and find that $\beta$-DARTS always achieves the best result among all compared NAS methods under almost all proxies. We further introduce the novel flooding regularization to the weight optimization of $\beta$-DARTS (i.e., Bi-level regularization), and experimentally and theoretically verify its effectiveness for improving the proxy robustness of differentiable NAS.

Related papers

• The devil is in discretization discrepancy. Robustifying Differentiable NAS with Single-Stage Searching Protocol [2.4300749758571905]
  gradient-based methods suffer from the discretization error, which can severely damage the process of obtaining the final architecture. We introduce a novel single-stage searching protocol, which is not reliant on decoding a continuous architecture. Our results demonstrate that this approach outperforms other DNAS methods by achieving 75.3% in the searching stage on the Cityscapes validation dataset.
  arXiv  Detail & Related papers  (2024-05-26T15:44:53Z)
• Generalizable Lightweight Proxy for Robust NAS against Diverse Perturbations [59.683234126055694]
  Recent neural architecture search (NAS) frameworks have been successful in finding optimal architectures for given conditions. We propose a novel lightweight robust zero-cost proxy that considers the consistency across features, parameters, and gradients of both clean and perturbed images. Our approach facilitates an efficient and rapid search for neural architectures capable of learning generalizable features that exhibit robustness across diverse perturbations.
  arXiv  Detail & Related papers  (2023-06-08T08:34:26Z)
• $\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture Search [85.84110365657455]
  We propose a simple-but-efficient regularization method, termed as Beta-Decay, to regularize the DARTS-based NAS searching process. Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets.
  arXiv  Detail & Related papers  (2022-03-03T11:47:14Z)
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule [95.56873042777316]
  Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost. This paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions. We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability.
  arXiv  Detail & Related papers  (2021-11-25T18:13:42Z)
• Zero-Cost Proxies Meet Differentiable Architecture Search [20.957570100784988]
  Differentiable neural architecture search (NAS) has attracted significant attention in recent years. Despite its success, DARTS lacks robustness in certain cases. We propose a novel operation selection paradigm in the context of differentiable NAS.
  arXiv  Detail & Related papers  (2021-06-12T15:33:36Z)
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization [99.81980366552408]
  We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. We propose a perturbation-based regularization - SmoothDARTS (SDARTS) - to smooth the loss landscape and improve the generalizability of DARTS-based methods.
  arXiv  Detail & Related papers  (2020-02-12T23:46:58Z)
• EcoNAS: Finding Proxies for Economical Neural Architecture Search [130.59673917196994]
  In this paper, we observe that most existing proxies exhibit different behaviors in maintaining the rank consistency among network candidates. Inspired by these observations, we present a reliable proxy and further formulate a hierarchical proxy strategy. The strategy spends more computations on candidate networks that are potentially more accurate, while discards unpromising ones in early stage with a fast proxy.
  arXiv  Detail & Related papers  (2020-01-05T13:29:02Z)
• NAS evaluation is frustratingly hard [1.7188280334580197]
  Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. Comparison between different methods is still very much an open issue. Our first contribution is a benchmark of $8$ NAS methods on $5$ datasets.
  arXiv  Detail & Related papers  (2019-12-28T21:24:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.