ResBuilder: Automated Learning of Depth with Residual Structures
- URL: http://arxiv.org/abs/2308.08504v1
- Date: Wed, 16 Aug 2023 16:58:25 GMT
- Title: ResBuilder: Automated Learning of Depth with Residual Structures
- Authors: Julian Burghoff, Matthias Rottmann, Jill von Conta, Sebastian
Schoenen, Andreas Witte, Hanno Gottschalk
- Abstract summary: We develop a neural architecture search algorithm, termed Resbuilder, that develops ResNet architectures from scratch.
Resbuilder achieves close to state-of-the-art performance while saving computational cost compared to off-the-shelf ResNets.
We demonstrate that this property generalizes even to industrial applications by applying our method with default parameters on a proprietary fraud detection dataset.
- Score: 5.172964916120902
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this work, we develop a neural architecture search algorithm, termed
Resbuilder, that develops ResNet architectures from scratch that achieve high
accuracy at moderate computational cost. It can also be used to modify existing
architectures and has the capability to remove and insert ResNet blocks, in
this way searching for suitable architectures in the space of ResNet
architectures. In our experiments on different image classification datasets,
Resbuilder achieves close to state-of-the-art performance while saving
computational cost compared to off-the-shelf ResNets. Noteworthy, we once tune
the parameters on CIFAR10 which yields a suitable default choice for all other
datasets. We demonstrate that this property generalizes even to industrial
applications by applying our method with default parameters on a proprietary
fraud detection dataset.
Related papers
- Mechanistic Design and Scaling of Hybrid Architectures [114.3129802943915]
We identify and test new hybrid architectures constructed from a variety of computational primitives.
We experimentally validate the resulting architectures via an extensive compute-optimal and a new state-optimal scaling law analysis.
We find MAD synthetics to correlate with compute-optimal perplexity, enabling accurate evaluation of new architectures.
arXiv Detail & Related papers (2024-03-26T16:33:12Z) - Building Optimal Neural Architectures using Interpretable Knowledge [15.66288233048004]
AutoBuild is a scheme which learns to align the latent embeddings of operations and architecture modules with the ground-truth performance of the architectures they appear in.
We show that by mining a relatively small set of evaluated architectures, AutoBuild can learn to build high-quality architectures directly or help to reduce search space to focus on relevant areas.
arXiv Detail & Related papers (2024-03-20T04:18:38Z) - FlowNAS: Neural Architecture Search for Optical Flow Estimation [65.44079917247369]
We propose a neural architecture search method named FlowNAS to automatically find the better encoder architecture for flow estimation task.
Experimental results show that the discovered architecture with the weights inherited from the super-network achieves 4.67% F1-all error on KITTI.
arXiv Detail & Related papers (2022-07-04T09:05:25Z) - De-IReps: Searching for improved Re-parameterizing Architecture based on
Differentiable Evolution Strategy [5.495046508448319]
We design a search space that covers almost all re- parameterization operations.
In this search space, multiple-path networks can be unconditionally re- parameterized into single-path networks.
We visualize the features of the searched architecture and give our explanation for the appearance of this architecture.
arXiv Detail & Related papers (2022-04-13T14:07:20Z) - Learning Interpretable Models Through Multi-Objective Neural
Architecture Search [0.9990687944474739]
We propose a framework to optimize for both task performance and "introspectability," a surrogate metric for aspects of interpretability.
We demonstrate that jointly optimizing for task error and introspectability leads to more disentangled and debuggable architectures that perform within error.
arXiv Detail & Related papers (2021-12-16T05:50:55Z) - Network Graph Based Neural Architecture Search [57.78724765340237]
We search neural network by rewiring the corresponding graph and predict the architecture performance by graph properties.
Because we do not perform machine learning over the entire graph space, the searching process is remarkably efficient.
arXiv Detail & Related papers (2021-12-15T00:12:03Z) - Rethinking Architecture Selection in Differentiable NAS [74.61723678821049]
Differentiable Neural Architecture Search is one of the most popular NAS methods for its search efficiency and simplicity.
We propose an alternative perturbation-based architecture selection that directly measures each operation's influence on the supernet.
We find that several failure modes of DARTS can be greatly alleviated with the proposed selection method.
arXiv Detail & Related papers (2021-08-10T00:53:39Z) - Neighborhood-Aware Neural Architecture Search [43.87465987957761]
We propose a novel neural architecture search (NAS) method to identify flat-minima architectures in the search space.
Our formulation takes the "flatness" of an architecture into account by aggregating the performance over the neighborhood of this architecture.
Based on our formulation, we propose neighborhood-aware random search (NA-RS) and neighborhood-aware differentiable architecture search (NA-DARTS)
arXiv Detail & Related papers (2021-05-13T15:56:52Z) - Operation Embeddings for Neural Architecture Search [15.033712726016255]
We propose the replacement of fixed operator encoding with learnable representations in the optimization process.
Our method produces top-performing architectures that share similar operation and graph patterns.
arXiv Detail & Related papers (2021-05-11T09:17:10Z) - A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
We employ an auto-encoder to discover meaningful representations of neural architectures.
A graph convolutional neural network is introduced to predict the performance of architectures.
arXiv Detail & Related papers (2020-05-14T09:02:33Z) - Stage-Wise Neural Architecture Search [65.03109178056937]
Modern convolutional networks such as ResNet and NASNet have achieved state-of-the-art results in many computer vision applications.
These networks consist of stages, which are sets of layers that operate on representations in the same resolution.
It has been demonstrated that increasing the number of layers in each stage improves the prediction ability of the network.
However, the resulting architecture becomes computationally expensive in terms of floating point operations, memory requirements and inference time.
arXiv Detail & Related papers (2020-04-23T14:16:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.