Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces
- URL: http://arxiv.org/abs/2012.08859v1
- Date: Wed, 16 Dec 2020 11:00:19 GMT
- Title: Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces
- Authors: Bert Moons, Parham Noorzad, Andrii Skliar, Giovanni Mariani, Dushyant
Mehta, Chris Lott, Tijmen Blankevoort
- Abstract summary: DONNA (Distilling Optimal Neural Network Architectures) is a novel pipeline for rapid neural architecture search and search space exploration.
In ImageNet classification, architectures found by DONNA are 20% faster than EfficientNet-B0 and MobileNetV2 on a Nvidia V100 GPU at similar accuracy and 10% faster with 0.5% higher accuracy than MobileNetV2-1.4x on a Samsung S20 smartphone.
- Score: 16.920328058816338
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This work presents DONNA (Distilling Optimal Neural Network Architectures), a
novel pipeline for rapid neural architecture search and search space
exploration, targeting multiple different hardware platforms and user
scenarios. In DONNA, a search consists of three phases. First, an accuracy
predictor is built for a diverse search space using blockwise knowledge
distillation. This predictor enables searching across diverse
macro-architectural network parameters such as layer types, attention
mechanisms, and channel widths, as well as across micro-architectural
parameters such as block repeats, kernel sizes, and expansion rates. Second, a
rapid evolutionary search phase finds a Pareto-optimal set of architectures in
terms of accuracy and latency for any scenario using the predictor and
on-device measurements. Third, Pareto-optimal models can be quickly finetuned
to full accuracy. With this approach, DONNA finds architectures that outperform
the state of the art. In ImageNet classification, architectures found by DONNA
are 20% faster than EfficientNet-B0 and MobileNetV2 on a Nvidia V100 GPU at
similar accuracy and 10% faster with 0.5% higher accuracy than MobileNetV2-1.4x
on a Samsung S20 smartphone. In addition to neural architecture search, DONNA
is used for search-space exploration and hardware-aware model compression.
Related papers
- Flexible Channel Dimensions for Differentiable Architecture Search [50.33956216274694]
We propose a novel differentiable neural architecture search method with an efficient dynamic channel allocation algorithm.
We show that the proposed framework is able to find DNN architectures that are equivalent to previous methods in task accuracy and inference latency.
arXiv Detail & Related papers (2023-06-13T15:21:38Z) - DASS: Differentiable Architecture Search for Sparse neural networks [0.5735035463793009]
We find that the architectures designed for dense networks by differentiable architecture search methods are ineffective when pruning mechanisms are applied to them.
In this paper, we propose a new method to search for sparsity-friendly neural architectures.
We do this by adding two new sparse operations to the search space and modifying the search objective.
arXiv Detail & Related papers (2022-07-14T14:53:50Z) - Profiling Neural Blocks and Design Spaces for Mobile Neural Architecture
Search [21.48915618572691]
We analyze the neural blocks used to build Once-for-All (MobileNetV3), ProxylessNAS and ResNet families.
We show that searching in the reduced search space generates better accuracy-latency frontiers than searching in the original search spaces.
arXiv Detail & Related papers (2021-09-25T19:34:45Z) - MS-RANAS: Multi-Scale Resource-Aware Neural Architecture Search [94.80212602202518]
We propose Multi-Scale Resource-Aware Neural Architecture Search (MS-RANAS)
We employ a one-shot architecture search approach in order to obtain a reduced search cost.
We achieve state-of-the-art results in terms of accuracy-speed trade-off.
arXiv Detail & Related papers (2020-09-29T11:56:01Z) - FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining [65.39532971991778]
We present an accuracy predictor that scores architecture and training recipes jointly, guiding both sample selection and ranking.
We run fast evolutionary searches in just CPU minutes to generate architecture-recipe pairs for a variety of resource constraints.
FBNetV3 makes up a family of state-of-the-art compact neural networks that outperform both automatically and manually-designed competitors.
arXiv Detail & Related papers (2020-06-03T05:20:21Z) - FBNetV2: Differentiable Neural Architecture Search for Spatial and
Channel Dimensions [70.59851564292828]
Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks.
We propose a memory and computationally efficient DNAS variant: DMaskingNAS.
This algorithm expands the search space by up to $1014times$ over conventional DNAS.
arXiv Detail & Related papers (2020-04-12T08:52:15Z) - Latency-Aware Differentiable Neural Architecture Search [113.35689580508343]
Differentiable neural architecture search methods became popular in recent years, mainly due to their low search costs and flexibility in designing the search space.
However, these methods suffer the difficulty in optimizing network, so that the searched network is often unfriendly to hardware.
This paper deals with this problem by adding a differentiable latency loss term into optimization, so that the search process can tradeoff between accuracy and latency with a balancing coefficient.
arXiv Detail & Related papers (2020-01-17T15:55:21Z) - DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution
Pruning [135.27931587381596]
We propose an efficient and unified NAS framework termed DDPNAS via dynamic distribution pruning.
In particular, we first sample architectures from a joint categorical distribution. Then the search space is dynamically pruned and its distribution is updated every few epochs.
With the proposed efficient network generation method, we directly obtain the optimal neural architectures on given constraints.
arXiv Detail & Related papers (2019-05-28T06:35:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.