Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search

• URL: http://arxiv.org/abs/2304.05405v2
• Date: Mon, 1 May 2023 02:26:52 GMT
• Title: Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search
• Authors: Alexandre Heuillet, Ahmad Nasser, Hichem Arioui, Hedi Tabia
• Abstract summary: Differentiable Neural Architecture Search (DNAS) rapidly imposed itself as the trending approach to automate the discovery of deep neural network architectures. This rise is mainly due to the popularity of DARTS, one of the first major DNAS methods. In this comprehensive survey, we focus specifically on DNAS and review recent approaches in this field.
• Score: 70.31239620427526
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the past few years, Differentiable Neural Architecture Search (DNAS) rapidly imposed itself as the trending approach to automate the discovery of deep neural network architectures. This rise is mainly due to the popularity of DARTS, one of the first major DNAS methods. In contrast with previous works based on Reinforcement Learning or Evolutionary Algorithms, DNAS is faster by several orders of magnitude and uses fewer computational resources. In this comprehensive survey, we focus specifically on DNAS and review recent approaches in this field. Furthermore, we propose a novel challenge-based taxonomy to classify DNAS methods. We also discuss the contributions brought to DNAS in the past few years and its impact on the global NAS field. Finally, we conclude by giving some insights into future research directions for the DNAS field.

Related papers

• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• A Kernel-Based Neural Network Test for High-dimensional Sequencing Data Analysis [0.8221435109014762]
  We introduce a new kernel-based neural network (KNN) test for complex association analysis of sequencing data. Based on KNN, a Wald-type test is then introduced to evaluate the joint association of high-dimensional genetic data with a disease phenotype of interest.
  arXiv  Detail & Related papers  (2023-12-05T16:06:23Z)
• Accelerating Neural Architecture Exploration Across Modalities Using Genetic Algorithms [5.620334754517149]
  We show how genetic algorithms can be paired with lightly trained objective predictors in an iterative cycle to accelerate multi-objective architectural exploration. NAS research efforts have centered around computer vision tasks and only recently have other modalities, such as the rapidly growing field of natural language processing, been investigated in depth.
  arXiv  Detail & Related papers  (2022-02-25T20:01:36Z)
• Neural Architecture Search for Dense Prediction Tasks in Computer Vision [74.9839082859151]
  Deep learning has led to a rising demand for neural network architecture engineering. neural architecture search (NAS) aims at automatically designing neural network architectures in a data-driven manner rather than manually. NAS has become applicable to a much wider range of problems in computer vision.
  arXiv  Detail & Related papers  (2022-02-15T08:06:50Z)
• Differentiable NAS Framework and Application to Ads CTR Prediction [30.74403362212425]
  We implement an inference and modular framework for Differentiable Neural Architecture Search (DNAS) We apply DNAS to the problem of ads click-through rate (CTR) prediction, arguably the highest-value and most worked on AI problem at hyperscalers today. We develop and tailor novel search spaces to a Deep Learning Recommendation Model (DLRM) backbone for CTR prediction, and report state-of-the-art results on the Criteo Kaggle CTR prediction dataset.
  arXiv  Detail & Related papers  (2021-10-25T05:46:27Z)
• Improving the sample-efficiency of neural architecture search with reinforcement learning [0.0]
  In this work, we would like to contribute to the area of Automated Machine Learning (AutoML) Our focus is on one of the most promising research directions, reinforcement learning. The validation accuracies of the child networks serve as a reward signal for training the controller. We propose to modify this to a more modern and complex algorithm, PPO, which has demonstrated to be faster and more stable in other environments.
  arXiv  Detail & Related papers  (2021-10-13T14:30:09Z)
• D-DARTS: Distributed Differentiable Architecture Search [75.12821786565318]
  Differentiable ARchiTecture Search (DARTS) is one of the most trending Neural Architecture Search (NAS) methods. We propose D-DARTS, a novel solution that addresses this problem by nesting several neural networks at cell-level.
  arXiv  Detail & Related papers  (2021-08-20T09:07:01Z)
• On the Exploitation of Neuroevolutionary Information: Analyzing the Past for a More Efficient Future [60.99717891994599]
  We propose an approach that extracts information from neuroevolutionary runs, and use it to build a metamodel. We inspect the best structures found during neuroevolutionary searches of generative adversarial networks with varying characteristics.
  arXiv  Detail & Related papers  (2021-05-26T20:55:29Z)
• Search to aggregate neighborhood for graph neural network [47.47628113034479]
  We propose a framework, which tries to Search to Aggregate NEighborhood (SANE) to automatically design data-specific GNN architectures. By designing a novel and expressive search space, we propose a differentiable search algorithm, which is more efficient than previous reinforcement learning based methods.
  arXiv  Detail & Related papers  (2021-04-14T03:15:19Z)
• NAS-Navigator: Visual Steering for Explainable One-Shot Deep Neural Network Synthesis [53.106414896248246]
  We present a framework that allows analysts to effectively build the solution sub-graph space and guide the network search by injecting their domain knowledge. Applying this technique in an iterative manner allows analysts to converge to the best performing neural network architecture for a given application.
  arXiv  Detail & Related papers  (2020-09-28T01:48:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.