MS-DARTS: Mean-Shift Based Differentiable Architecture Search

• URL: http://arxiv.org/abs/2108.09996v1
• Date: Mon, 23 Aug 2021 08:06:45 GMT
• Title: MS-DARTS: Mean-Shift Based Differentiable Architecture Search
• Authors: Jun-Wei Hsieh, Ming-Ching Chang, Ping-Yang Chen, Cheng-Han Chou, Chih-Sheng Huang
• Abstract summary: We propose a Mean-Shift based DARTS (MS-DARTS) to improve stability based on sampling and perturbation. MS-DARTS archives higher performance over other state-of-the-art NAS methods with reduced search cost.
• Score: 11.115656548869199
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Differentiable Architecture Search (DARTS) is an effective continuous relaxation-based network architecture search (NAS) method with low search cost. It has attracted significant attentions in Auto-ML research and becomes one of the most useful paradigms in NAS. Although DARTS can produce superior efficiency over traditional NAS approaches with better control of complex parameters, oftentimes it suffers from stabilization issues in producing deteriorating architectures when discretizing the continuous architecture. We observed considerable loss of validity causing dramatic decline in performance at this final discretization step of DARTS. To address this issue, we propose a Mean-Shift based DARTS (MS-DARTS) to improve stability based on sampling and perturbation. Our approach can improve bot the stability and accuracy of DARTS, by smoothing the loss landscape and sampling architecture parameters within a suitable bandwidth. We investigate the convergence of our mean-shift approach, together with the effects of bandwidth selection that affects stability and accuracy. Evaluations performed on CIFAR-10, CIFAR-100, and ImageNet show that MS-DARTS archives higher performance over other state-of-the-art NAS methods with reduced search cost.

Related papers

• OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strength [70.76342136866413]
  Differentiable architecture search (DARTS) has emerged as a promising technique for effective neural architecture search. DARTS suffers from the well-known degeneration issue which can lead to deteriorating architectures. We propose a novel criterion based on operation strength that estimates the importance of an operation by its effect on the final loss.
  arXiv  Detail & Related papers  (2024-09-22T13:16:07Z)
• The devil is in discretization discrepancy. Robustifying Differentiable NAS with Single-Stage Searching Protocol [2.4300749758571905]
  gradient-based methods suffer from the discretization error, which can severely damage the process of obtaining the final architecture. We introduce a novel single-stage searching protocol, which is not reliant on decoding a continuous architecture. Our results demonstrate that this approach outperforms other DNAS methods by achieving 75.3% in the searching stage on the Cityscapes validation dataset.
  arXiv  Detail & Related papers  (2024-05-26T15:44:53Z)
• IS-DARTS: Stabilizing DARTS through Precise Measurement on Candidate Importance [41.23462863659102]
  DARTS is known for its efficiency and simplicity. However, performance collapse in DARTS results in deteriorating architectures filled with parameter-free operations. We propose IS-DARTS to comprehensively improve DARTS and resolve the aforementioned problems.
  arXiv  Detail & Related papers  (2023-12-19T22:45:57Z)
• Enhancing the Robustness, Efficiency, and Diversity of Differentiable Architecture Search [25.112048502327738]
  Differentiable architecture search (DARTS) has attracted much attention due to its simplicity and significant improvement in efficiency. Many works attempt to restrict the accumulation of skip connections by indicators or manual design. We suggest a more subtle and direct approach that removes skip connections from the operation space.
  arXiv  Detail & Related papers  (2022-04-10T13:25:36Z)
• $\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture Search [85.84110365657455]
  We propose a simple-but-efficient regularization method, termed as Beta-Decay, to regularize the DARTS-based NAS searching process. Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets.
  arXiv  Detail & Related papers  (2022-03-03T11:47:14Z)
• ZARTS: On Zero-order Optimization for Neural Architecture Search [94.41017048659664]
  Differentiable architecture search (DARTS) has been a popular one-shot paradigm for NAS due to its high efficiency. This work turns to zero-order optimization and proposes a novel NAS scheme, called ZARTS, to search without enforcing the above approximation. In particular, results on 12 benchmarks verify the outstanding robustness of ZARTS, where the performance of DARTS collapses due to its known instability issue.
  arXiv  Detail & Related papers  (2021-10-10T09:35:15Z)
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization [51.489024258966886]
  Differentiable ARchiTecture Search (DARTS) uses a continuous relaxation of network representation and dramatically accelerates Neural Architecture Search (NAS) by almost thousands of times in GPU-day. However, the searching process of DARTS is unstable, which suffers severe degradation when training epochs become large. We propose an improved version of DARTS, namely iDARTS, to deal with the two problems.
  arXiv  Detail & Related papers  (2021-08-25T02:23:30Z)
• iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients [75.41173109807735]
  Differentiable ARchiTecture Search (DARTS) has recently become the mainstream of neural architecture search (NAS) We tackle the hypergradient computation in DARTS based on the implicit function theorem. We show that the architecture optimisation with the proposed method, named iDARTS, is expected to converge to a stationary point.
  arXiv  Detail & Related papers  (2021-06-21T00:44:11Z)
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization [99.81980366552408]
  We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. We propose a perturbation-based regularization - SmoothDARTS (SDARTS) - to smooth the loss landscape and improve the generalizability of DARTS-based methods.
  arXiv  Detail & Related papers  (2020-02-12T23:46:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.