Noisy Differentiable Architecture Search

• URL: http://arxiv.org/abs/2005.03566v3
• Date: Sun, 17 Oct 2021 14:57:46 GMT
• Title: Noisy Differentiable Architecture Search
• Authors: Xiangxiang Chu and Bo Zhang
• Abstract summary: Differentiable Architecture Search (DARTS) has now become one of the mainstream paradigms of neural architecture search. It largely suffers from the well-known performance collapse issue due to the aggregation of skip connections. We propose to inject unbiased random noise to impede the flow.
• Score: 13.154295073267367
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simplicity is the ultimate sophistication. Differentiable Architecture Search (DARTS) has now become one of the mainstream paradigms of neural architecture search. However, it largely suffers from the well-known performance collapse issue due to the aggregation of skip connections. It is thought to have overly benefited from the residual structure which accelerates the information flow. To weaken this impact, we propose to inject unbiased random noise to impede the flow. We name this novel approach NoisyDARTS. In effect, a network optimizer should perceive this difficulty at each training step and refrain from overshooting, especially on skip connections. In the long run, since we add no bias to the gradient in terms of expectation, it is still likely to converge to the right solution area. We also prove that the injected noise plays a role in smoothing the loss landscape, which makes the optimization easier. Our method features extreme simplicity and acts as a new strong baseline. We perform extensive experiments across various search spaces, datasets, and tasks, where we robustly achieve state-of-the-art results. Our code is available at https://github.com/xiaomi-automl/NoisyDARTS.

Related papers

• Lightweight Diffusion Models with Distillation-Based Block Neural Architecture Search [55.41583104734349]
  We propose to automatically remove structural redundancy in diffusion models with our proposed Diffusion Distillation-based Block-wise Neural Architecture Search (NAS) Given a larger pretrained teacher, we leverage DiffNAS to search for the smallest architecture which can achieve on-par or even better performance than the teacher. Different from previous block-wise NAS methods, DiffNAS contains a block-wise local search strategy and a retraining strategy with a joint dynamic loss.
  arXiv  Detail & Related papers  (2023-11-08T12:56:59Z)
• Operation-level Progressive Differentiable Architecture Search [19.214462477848535]
  We propose operation-level progressive differentiable neural architecture search (OPP-DARTS) to avoid skip connections aggregation. Our method's performance on CIFAR-10 is superior to the architecture found by standard DARTS.
  arXiv  Detail & Related papers  (2023-02-11T09:18:01Z)
• Enhancing the Robustness, Efficiency, and Diversity of Differentiable Architecture Search [25.112048502327738]
  Differentiable architecture search (DARTS) has attracted much attention due to its simplicity and significant improvement in efficiency. Many works attempt to restrict the accumulation of skip connections by indicators or manual design. We suggest a more subtle and direct approach that removes skip connections from the operation space.
  arXiv  Detail & Related papers  (2022-04-10T13:25:36Z)
• Efficient Person Search: An Anchor-Free Approach [86.45858994806471]
  Person search aims to simultaneously localize and identify a query person from realistic, uncropped images. To achieve this goal, state-of-the-art models typically add a re-id branch upon two-stage detectors like Faster R-CNN. In this work, we present an anchor-free approach to efficiently tackling this challenging task, by introducing the following dedicated designs.
  arXiv  Detail & Related papers  (2021-09-01T07:01:33Z)
• D-DARTS: Distributed Differentiable Architecture Search [75.12821786565318]
  Differentiable ARchiTecture Search (DARTS) is one of the most trending Neural Architecture Search (NAS) methods. We propose D-DARTS, a novel solution that addresses this problem by nesting several neural networks at cell-level.
  arXiv  Detail & Related papers  (2021-08-20T09:07:01Z)
• Neural Architecture Search via Bregman Iterations [1.7778609937758323]
  We propose a novel strategy for Neural Architecture Search (NAS) based on Bregman iterations. Our gradient-based one-shot algorithm gradually adds relevant parameters in an inverse scale space manner.
  arXiv  Detail & Related papers  (2021-06-04T13:37:47Z)
• Making Differentiable Architecture Search less local [9.869449181400466]
  Differentiable neural architecture search (DARTS) is a promising NAS approach that dramatically increases search efficiency. It has been shown to suffer from performance collapse, where the search often leads to detrimental architectures. We develop a more global optimisation scheme that is able to better explore the space without changing the DARTS problem formulation.
  arXiv  Detail & Related papers  (2021-04-21T10:36:43Z)
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search [60.965024145243596]
  One-shot weight sharing methods have recently drawn great attention in neural architecture search due to high efficiency and competitive performance. To alleviate this problem, we present a simple yet effective architecture distillation method. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop.
  arXiv  Detail & Related papers  (2020-10-29T17:55:05Z)
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators [74.21019737169675]
  Differentiable architecture search suffers from long-standing performance instability. indicators such as Hessian eigenvalues are proposed as a signal to stop searching before the performance collapses. In this paper, we undertake a more subtle and direct approach to resolve the collapse.
  arXiv  Detail & Related papers  (2020-09-02T12:54:13Z)
• Latency-Aware Differentiable Neural Architecture Search [113.35689580508343]
  Differentiable neural architecture search methods became popular in recent years, mainly due to their low search costs and flexibility in designing the search space. However, these methods suffer the difficulty in optimizing network, so that the searched network is often unfriendly to hardware. This paper deals with this problem by adding a differentiable latency loss term into optimization, so that the search process can tradeoff between accuracy and latency with a balancing coefficient.
  arXiv  Detail & Related papers  (2020-01-17T15:55:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.