Analyze and Design Network Architectures by Recursion Formulas

• URL: http://arxiv.org/abs/2108.08689v1
• Date: Wed, 18 Aug 2021 06:53:30 GMT
• Title: Analyze and Design Network Architectures by Recursion Formulas
• Authors: Yilin Liao, Hao Wang, Zhaoran Liu, Haozhe Li and Xinggao Liu
• Abstract summary: This work attempts to find an effective way to design new network architectures. It is discovered that the main difference between network architectures can be reflected in their formulas. A case study is provided to generate an improved architecture based on ResNet. Massive experiments are conducted on CIFAR and ImageNet, which witness the significant performance improvements.
• Score: 4.085771561472743
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The effectiveness of shortcut/skip-connection has been widely verified, which inspires massive explorations on neural architecture design. This work attempts to find an effective way to design new network architectures. It is discovered that the main difference between network architectures can be reflected in their recursion formulas. Based on this, a methodology is proposed to design novel network architectures from the perspective of mathematical formulas. Afterwards, a case study is provided to generate an improved architecture based on ResNet. Furthermore, the new architecture is compared with ResNet and then tested on ResNet-based networks. Massive experiments are conducted on CIFAR and ImageNet, which witnesses the significant performance improvements provided by the architecture.

Related papers

• Designing Network Design Strategies Through Gradient Path Analysis [12.90962626557934]
  This paper proposes a new network design strategy, i.e., to design the network architecture based on gradient path analysis. We propose the gradient path design strategies for the layer-level, the stage-level, and the network-level.
  arXiv  Detail & Related papers  (2022-11-09T10:51:57Z)
• Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles [22.66053583920441]
  ArchExplorer is a visual analysis method for understanding a neural architecture space and summarizing design principles. A circle-packing-based architecture visualization has been developed to convey both the global relationships between clusters and local neighborhoods of the architectures in each cluster. Two case studies and a post-analysis are presented to demonstrate the effectiveness of ArchExplorer in summarizing design principles and selecting better-performing architectures.
  arXiv  Detail & Related papers  (2022-08-20T12:15:59Z)
• Simple and Efficient Architectures for Semantic Segmentation [50.1563637917129]
  We show that a simple encoder-decoder architecture with a ResNet-like backbone and a small multi-scale head, performs on-par or better than complex semantic segmentation architectures such as HRNet, FANet and DDRNet. We present a family of such simple architectures for desktop as well as mobile targets, which match or exceed the performance of complex models on the Cityscapes dataset.
  arXiv  Detail & Related papers  (2022-06-16T15:08:34Z)
• Hysteretic Behavior Simulation Based on Pyramid Neural Network:Principle, Network Architecture, Case Study and Explanation [0.0]
  A surrogate model based on neural networks shows significant potential in balancing efficiency and accuracy. Its serial information flow and prediction based on single-level features adversely affect the network performance. A weighted stacked pyramid neural network architecture is proposed herein.
  arXiv  Detail & Related papers  (2022-04-29T16:42:00Z)
• Learning Interpretable Models Through Multi-Objective Neural Architecture Search [0.9990687944474739]
  We propose a framework to optimize for both task performance and "introspectability," a surrogate metric for aspects of interpretability. We demonstrate that jointly optimizing for task error and introspectability leads to more disentangled and debuggable architectures that perform within error.
  arXiv  Detail & Related papers  (2021-12-16T05:50:55Z)
• Network Graph Based Neural Architecture Search [57.78724765340237]
  We search neural network by rewiring the corresponding graph and predict the architecture performance by graph properties. Because we do not perform machine learning over the entire graph space, the searching process is remarkably efficient.
  arXiv  Detail & Related papers  (2021-12-15T00:12:03Z)
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements [11.978082858160576]
  Backbone architectures of most binary networks are well-known floating point (FP) architectures such as the ResNet family. We propose to search architectures for binary networks by defining a new search space for binary architectures and a novel search objective. We show that our method searches architectures with stable training curves despite the quantization error inherent in binary networks.
  arXiv  Detail & Related papers  (2021-10-16T12:38:26Z)
• Rethinking Architecture Selection in Differentiable NAS [74.61723678821049]
  Differentiable Neural Architecture Search is one of the most popular NAS methods for its search efficiency and simplicity. We propose an alternative perturbation-based architecture selection that directly measures each operation's influence on the supernet. We find that several failure modes of DARTS can be greatly alleviated with the proposed selection method.
  arXiv  Detail & Related papers  (2021-08-10T00:53:39Z)
• NAS-DIP: Learning Deep Image Prior with Neural Architecture Search [65.79109790446257]
  Recent work has shown that the structure of deep convolutional neural networks can be used as a structured image prior. We propose to search for neural architectures that capture stronger image priors. We search for an improved network by leveraging an existing neural architecture search algorithm.
  arXiv  Detail & Related papers  (2020-08-26T17:59:36Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)
• Stage-Wise Neural Architecture Search [65.03109178056937]
  Modern convolutional networks such as ResNet and NASNet have achieved state-of-the-art results in many computer vision applications. These networks consist of stages, which are sets of layers that operate on representations in the same resolution. It has been demonstrated that increasing the number of layers in each stage improves the prediction ability of the network. However, the resulting architecture becomes computationally expensive in terms of floating point operations, memory requirements and inference time.
  arXiv  Detail & Related papers  (2020-04-23T14:16:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.