Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars

• URL: http://arxiv.org/abs/2211.01842v3
• Date: Fri, 8 Dec 2023 14:09:11 GMT
• Title: Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars
• Authors: Simon Schrodi, Danny Stoll, Binxin Ru, Rhea Sukthanker, Thomas Brox, Frank Hutter
• Abstract summary: We introduce a unifying search space design framework based on context-free grammars. By enhancing and using their properties, we effectively enable search over the complete architecture. We show that our search strategy can be superior to existing Neural Architecture Search approaches.
• Score: 66.05096551112932
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards this goal but lack a unifying search space design framework and typically only search over some limited aspect of architectures. In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature. By enhancing and using their properties, we effectively enable search over the complete architecture and can foster regularity. Further, we propose an efficient hierarchical kernel design for a Bayesian Optimization search strategy to efficiently search over such huge spaces. We demonstrate the versatility of our search space design framework and show that our search strategy can be superior to existing NAS approaches. Code is available at https://github.com/automl/hierarchical_nas_construction.

Related papers

• Efficient NAS with FaDE on Hierarchical Spaces [0.6372911857214884]
  We present FaDE which uses differentiable architecture search to obtain relative performance predictions on finite regions of a hierarchical NAS space. FaDE is especially suited on deep hierarchical, respectively multi-cell search spaces.
  arXiv  Detail & Related papers  (2024-04-24T21:33:17Z)
• Automated Dominative Subspace Mining for Efficient Neural Architecture Search [36.06889021273405]
  We propose a novel Neural Architecture Search method via Dominative Subspace Mining (DSM-NAS) DSM-NAS finds promising architectures in automatically mined subspaces. Experimental results demonstrate that DSM-NAS not only reduces the search cost but also discovers better architectures than state-of-the-art methods in various benchmark search spaces.
  arXiv  Detail & Related papers  (2022-10-31T09:54:28Z)
• Searching a High-Performance Feature Extractor for Text Recognition Network [92.12492627169108]
  We design a domain-specific search space by exploring principles for having good feature extractors. As the space is huge and complexly structured, no existing NAS algorithms can be applied. We propose a two-stage algorithm to effectively search in the space.
  arXiv  Detail & Related papers  (2022-09-27T03:49:04Z)
• Towards Less Constrained Macro-Neural Architecture Search [2.685668802278155]
  Neural Architecture Search (NAS) networks achieve state-of-the-art performance in a variety of tasks. Most NAS methods rely heavily on human-defined assumptions that constrain the search. We present experiments showing that LCMNAS generates state-of-the-art architectures from scratch with minimal GPU computation.
  arXiv  Detail & Related papers  (2022-03-10T17:53:03Z)
• Exploring Complicated Search Spaces with Interleaving-Free Sampling [127.07551427957362]
  In this paper, we build the search algorithm upon a complicated search space with long-distance connections. We present a simple yet effective algorithm named textbfIF-NAS, where we perform a periodic sampling strategy to construct different sub-networks. In the proposed search space, IF-NAS outperform both random sampling and previous weight-sharing search algorithms by a significant margin.
  arXiv  Detail & Related papers  (2021-12-05T06:42:48Z)
• Unchain the Search Space with Hierarchical Differentiable Architecture Search [42.32368267716705]
  DAS-based methods mainly focus on searching for a repeatable cell structure, which is then stacked sequentially in multiple stages to form the networks. We propose a Hierarchical Differentiable Architecture Search (H-DAS) that performs architecture search both at the cell level and at the stage level. For the stage-level search, we systematically study the architectures of stages, including the number of cells in each stage and the connections between the cells.
  arXiv  Detail & Related papers  (2021-01-11T17:01:43Z)
• GOLD-NAS: Gradual, One-Level, Differentiable [100.12492801459105]
  We propose a novel algorithm named Gradual One-Level Differentiable Neural Architecture Search (GOLD-NAS) It introduces a variable resource constraint to one-level optimization so that the weak operators are gradually pruned out from the super-network.
  arXiv  Detail & Related papers  (2020-07-07T10:37:49Z)
• Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search [94.46818035655943]
  We propose a curriculum search method that starts from a small search space and gradually incorporates the learned knowledge to guide the search in a large space. With the proposed search strategy, our Curriculum Neural Architecture Search (CNAS) method significantly improves the search efficiency and finds better architectures than existing NAS methods.
  arXiv  Detail & Related papers  (2020-07-07T02:29:06Z)
• AutoSTR: Efficient Backbone Search for Scene Text Recognition [80.7290173000068]
  Scene text recognition (STR) is very challenging due to the diversity of text instances and the complexity of scenes. We propose automated STR (AutoSTR) to search data-dependent backbones to boost text recognition performance. Experiments demonstrate that, by searching data-dependent backbones, AutoSTR can outperform the state-of-the-art approaches on standard benchmarks.
  arXiv  Detail & Related papers  (2020-03-14T06:51:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.