IS-DARTS: Stabilizing DARTS through Precise Measurement on Candidate Importance

• URL: http://arxiv.org/abs/2312.12648v1
• Date: Tue, 19 Dec 2023 22:45:57 GMT
• Title: IS-DARTS: Stabilizing DARTS through Precise Measurement on Candidate Importance
• Authors: Hongyi He, Longjun Liu, Haonan Zhang and Nanning Zheng
• Abstract summary: DARTS is known for its efficiency and simplicity. However, performance collapse in DARTS results in deteriorating architectures filled with parameter-free operations. We propose IS-DARTS to comprehensively improve DARTS and resolve the aforementioned problems.
• Score: 41.23462863659102
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Among existing Neural Architecture Search methods, DARTS is known for its efficiency and simplicity. This approach applies continuous relaxation of network representation to construct a weight-sharing supernet and enables the identification of excellent subnets in just a few GPU days. However, performance collapse in DARTS results in deteriorating architectures filled with parameter-free operations and remains a great challenge to the robustness. To resolve this problem, we reveal that the fundamental reason is the biased estimation of the candidate importance in the search space through theoretical and experimental analysis, and more precisely select operations via information-based measurements. Furthermore, we demonstrate that the excessive concern over the supernet and inefficient utilization of data in bi-level optimization also account for suboptimal results. We adopt a more realistic objective focusing on the performance of subnets and simplify it with the help of the information-based measurements. Finally, we explain theoretically why progressively shrinking the width of the supernet is necessary and reduce the approximation error of optimal weights in DARTS. Our proposed method, named IS-DARTS, comprehensively improves DARTS and resolves the aforementioned problems. Extensive experiments on NAS-Bench-201 and DARTS-based search space demonstrate the effectiveness of IS-DARTS.

Related papers

• OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strength [70.76342136866413]
  Differentiable architecture search (DARTS) has emerged as a promising technique for effective neural architecture search. DARTS suffers from the well-known degeneration issue which can lead to deteriorating architectures. We propose a novel criterion based on operation strength that estimates the importance of an operation by its effect on the final loss.
  arXiv  Detail & Related papers  (2024-09-22T13:16:07Z)
• Efficient Architecture Search via Bi-level Data Pruning [70.29970746807882]
  This work pioneers an exploration into the critical role of dataset characteristics for DARTS bi-level optimization. We introduce a new progressive data pruning strategy that utilizes supernet prediction dynamics as the metric. Comprehensive evaluations on the NAS-Bench-201 search space, DARTS search space, and MobileNet-like search space validate that BDP reduces search costs by over 50%.
  arXiv  Detail & Related papers  (2023-12-21T02:48:44Z)
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization [8.93957397187611]
  Differentiable architecture search (DARTS) is a promising end to end NAS method. Recent studies cast doubt on the basic underlying hypotheses of DARTS. We propose a novel sparse-regularized approximation and an efficient mixed-sparsity training scheme to robustify DARTS.
  arXiv  Detail & Related papers  (2023-06-12T04:11:37Z)
• $\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture Search [85.84110365657455]
  We propose a simple-but-efficient regularization method, termed as Beta-Decay, to regularize the DARTS-based NAS searching process. Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets.
  arXiv  Detail & Related papers  (2022-03-03T11:47:14Z)
• DARTS without a Validation Set: Optimizing the Marginal Likelihood [33.26229536690996]
  Training-Speed-Estimate (TSE) has previously been used in place of the validation loss for gradient-based optimization in DARTS. We extend those results by applying various DARTS diagnostics and show several unusual behaviors arising from not using a validation set.
  arXiv  Detail & Related papers  (2021-12-24T10:16:38Z)
• MS-DARTS: Mean-Shift Based Differentiable Architecture Search [11.115656548869199]
  We propose a Mean-Shift based DARTS (MS-DARTS) to improve stability based on sampling and perturbation. MS-DARTS archives higher performance over other state-of-the-art NAS methods with reduced search cost.
  arXiv  Detail & Related papers  (2021-08-23T08:06:45Z)
• iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients [75.41173109807735]
  Differentiable ARchiTecture Search (DARTS) has recently become the mainstream of neural architecture search (NAS) We tackle the hypergradient computation in DARTS based on the implicit function theorem. We show that the architecture optimisation with the proposed method, named iDARTS, is expected to converge to a stationary point.
  arXiv  Detail & Related papers  (2021-06-21T00:44:11Z)
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization [99.81980366552408]
  We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. We propose a perturbation-based regularization - SmoothDARTS (SDARTS) - to smooth the loss landscape and improve the generalizability of DARTS-based methods.
  arXiv  Detail & Related papers  (2020-02-12T23:46:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.