Towards Less Constrained Macro-Neural Architecture Search

• URL: http://arxiv.org/abs/2203.05508v1
• Date: Thu, 10 Mar 2022 17:53:03 GMT
• Title: Towards Less Constrained Macro-Neural Architecture Search
• Authors: Vasco Lopes and Lu\'is A. Alexandre
• Abstract summary: Neural Architecture Search (NAS) networks achieve state-of-the-art performance in a variety of tasks. Most NAS methods rely heavily on human-defined assumptions that constrain the search. We present experiments showing that LCMNAS generates state-of-the-art architectures from scratch with minimal GPU computation.
• Score: 2.685668802278155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Networks found with Neural Architecture Search (NAS) achieve state-of-the-art performance in a variety of tasks, out-performing human-designed networks. However, most NAS methods heavily rely on human-defined assumptions that constrain the search: architecture's outer-skeletons, number of layers, parameter heuristics and search spaces. Additionally, common search spaces consist of repeatable modules (cells) instead of fully exploring the architecture's search space by designing entire architectures (macro-search). Imposing such constraints requires deep human expertise and restricts the search to pre-defined settings. In this paper, we propose LCMNAS, a method that pushes NAS to less constrained search spaces by performing macro-search without relying on pre-defined heuristics or bounded search spaces. LCMNAS introduces three components for the NAS pipeline: i) a method that leverages information about well-known architectures to autonomously generate complex search spaces based on Weighted Directed Graphs with hidden properties, ii) a evolutionary search strategy that generates complete architectures from scratch, and iii) a mixed-performance estimation approach that combines information about architectures at initialization stage and lower fidelity estimates to infer their trainability and capacity to model complex functions. We present experiments showing that LCMNAS generates state-of-the-art architectures from scratch with minimal GPU computation. We study the importance of different NAS components on a macro-search setting. Code for reproducibility is public at \url{https://github.com/VascoLopes/LCMNAS}.

Related papers

• einspace: Searching for Neural Architectures from Fundamental Operations [28.346238250052455]
  We introduce einspace, a search space based on a parameterised probabilistic context-free grammar. We show that competitive architectures can be obtained by searching from scratch, and we consistently find large improvements when initialising the search with strong baselines.
  arXiv  Detail & Related papers  (2024-05-31T14:25:45Z)
• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars [66.05096551112932]
  We introduce a unifying search space design framework based on context-free grammars. By enhancing and using their properties, we effectively enable search over the complete architecture. We show that our search strategy can be superior to existing Neural Architecture Search approaches.
  arXiv  Detail & Related papers  (2022-11-03T14:23:00Z)
• BLOX: Macro Neural Architecture Search Benchmark and Algorithms [16.296454205012733]
  Neural architecture search (NAS) has been successfully used to design numerous high-performance neural networks. NAS is typically compute-intensive, so most existing approaches restrict the search to decide the operations and topological structure of a single block only. Recent studies show that a macro search space, which allows blocks in a model to be different, can lead to better performance.
  arXiv  Detail & Related papers  (2022-10-13T18:06:39Z)
• On Redundancy and Diversity in Cell-based Neural Architecture Search [44.337381243798085]
  We conduct an empirical analysis of architectures from the popular cell-based search spaces. We find that the architecture performance is minimally sensitive to changes at large parts of the cells. By explicitly constraining cells to include these patterns, randomly sampled architectures can match or even outperform the state of the art.
  arXiv  Detail & Related papers  (2022-03-16T18:59:29Z)
• Memory-Efficient Hierarchical Neural Architecture Search for Image Restoration [68.6505473346005]
  We propose a memory-efficient hierarchical NAS HiNAS (HiNAS) for image denoising and image super-resolution tasks. With a single GTX1080Ti GPU, it takes only about 1 hour for searching for denoising network on BSD 500 and 3.5 hours for searching for the super-resolution structure on DIV2K.
  arXiv  Detail & Related papers  (2020-12-24T12:06:17Z)
• Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search [94.46818035655943]
  We propose a curriculum search method that starts from a small search space and gradually incorporates the learned knowledge to guide the search in a large space. With the proposed search strategy, our Curriculum Neural Architecture Search (CNAS) method significantly improves the search efficiency and finds better architectures than existing NAS methods.
  arXiv  Detail & Related papers  (2020-07-07T02:29:06Z)
• Learning Architectures from an Extended Search Space for Language Modeling [37.79977691127229]
  We present a general approach to learn both intra-cell and inter-cell architectures of Neural architecture search (NAS) For recurrent neural language modeling, it outperforms a strong baseline significantly on the PTB and WikiText data, with a new state-of-the-art on PTB. The learned architectures show good transferability to other systems.
  arXiv  Detail & Related papers  (2020-05-06T05:02:33Z)
• Angle-based Search Space Shrinking for Neural Architecture Search [78.49722661000442]
  Angle-Based search space Shrinking (ABS) for Neural Architecture Search (NAS) Our approach progressively simplifies the original search space by dropping unpromising candidates. ABS can dramatically enhance existing NAS approaches by providing a promising shrunk search space.
  arXiv  Detail & Related papers  (2020-04-28T11:26:46Z)
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning [71.90902837008278]
  We propose to incorporate neural architecture search (NAS) into general-purpose multi-task learning (GP-MTL) In order to adapt to different task combinations, we disentangle the GP-MTL networks into single-task backbones. We also propose a novel single-shot gradient-based search algorithm that closes the performance gap between the searched architectures.
  arXiv  Detail & Related papers  (2020-03-31T09:49:14Z)
• DCNAS: Densely Connected Neural Architecture Search for Semantic Image Segmentation [44.46852065566759]
  We propose a Densely Connected NAS (DCNAS) framework, which directly searches the optimal network structures for the multi-scale representations of visual information. Specifically, by connecting cells with each other using learnable weights, we introduce a densely connected search space to cover an abundance of mainstream network designs. We demonstrate that the architecture obtained from our DCNAS algorithm achieves state-of-the-art performances on public semantic image segmentation benchmarks.
  arXiv  Detail & Related papers  (2020-03-26T13:21:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.