Efficient Training Under Limited Resources
- URL: http://arxiv.org/abs/2301.09264v1
- Date: Mon, 23 Jan 2023 04:26:20 GMT
- Title: Efficient Training Under Limited Resources
- Authors: Mahdi Zolnouri, Dounia Lakhmiri, Christophe Tribes, Eyy\"ub Sari,
S\'ebastien Le Digabel
- Abstract summary: Training time budget and size of the dataset are among the factors affecting the performance of a Deep Neural Network (DNN)
We present our approach to achieving such a goal in three steps.
Our work achieved an accuracy of 86.0 % on a tiny subset of Mini-ImageNet at the ICLR 2021 Hardware Aware Efficient Training (HAET) Challenge.
- Score: 1.5749416770494706
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Training time budget and size of the dataset are among the factors affecting
the performance of a Deep Neural Network (DNN). This paper shows that Neural
Architecture Search (NAS), Hyper Parameters Optimization (HPO), and Data
Augmentation help DNNs perform much better while these two factors are limited.
However, searching for an optimal architecture and the best hyperparameter
values besides a good combination of data augmentation techniques under low
resources requires many experiments. We present our approach to achieving such
a goal in three steps: reducing training epoch time by compressing the model
while maintaining the performance compared to the original model, preventing
model overfitting when the dataset is small, and performing the hyperparameter
tuning. We used NOMAD, which is a blackbox optimization software based on a
derivative-free algorithm to do NAS and HPO. Our work achieved an accuracy of
86.0 % on a tiny subset of Mini-ImageNet at the ICLR 2021 Hardware Aware
Efficient Training (HAET) Challenge and won second place in the competition.
The competition results can be found at haet2021.github.io/challenge and our
source code can be found at github.com/DouniaLakhmiri/ICLR\_HAET2021.
Related papers
- Effective pruning of web-scale datasets based on complexity of concept
clusters [48.125618324485195]
We present a method for pruning large-scale multimodal datasets for training CLIP-style models on ImageNet.
We find that training on a smaller set of high-quality data can lead to higher performance with significantly lower training costs.
We achieve a new state-of-the-art Imagehttps://info.arxiv.org/help/prep#commentsNet zero-shot accuracy and a competitive average zero-shot accuracy on 38 evaluation tasks.
arXiv Detail & Related papers (2024-01-09T14:32:24Z) - Resource Constrained Model Compression via Minimax Optimization for
Spiking Neural Networks [11.19282454437627]
Spiking Neural Networks (SNNs) have the characteristics of event-driven and high energy-efficient networks.
It is difficult to deploy these networks on resource-limited edge devices directly.
We propose an improved end-to-end Minimax optimization method for this sparse learning problem.
arXiv Detail & Related papers (2023-08-09T02:50:15Z) - Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML [4.2019872499238256]
We propose a novel strategy for deploying Deep Neural Networks on microcontrollers (TinyML) based on Multi-Objective Bayesian optimization (MOBOpt)
Our methodology aims at efficiently finding tradeoffs between a DNN's predictive accuracy, memory consumption on a given target system, and computational complexity.
arXiv Detail & Related papers (2023-05-23T14:31:52Z) - Recurrent Bilinear Optimization for Binary Neural Networks [58.972212365275595]
BNNs neglect the intrinsic bilinear relationship of real-valued weights and scale factors.
Our work is the first attempt to optimize BNNs from the bilinear perspective.
We obtain robust RBONNs, which show impressive performance over state-of-the-art BNNs on various models and datasets.
arXiv Detail & Related papers (2022-09-04T06:45:33Z) - U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture
Search [50.33956216274694]
optimizing resource utilization in target platforms is key to achieving high performance during DNN inference.
We propose a novel hardware-aware NAS framework that does not only optimize for task accuracy and inference latency, but also for resource utilization.
We achieve 2.8 - 4x speedup for DNN inference compared to prior hardware-aware NAS methods.
arXiv Detail & Related papers (2022-03-23T13:44:15Z) - Accelerating Multi-Objective Neural Architecture Search by Random-Weight
Evaluation [24.44521525130034]
We introduce a new performance estimation metric named Random-Weight Evaluation (RWE) to quantify the quality of CNNs.
RWE only trains its last layer and leaves the remainders with randomly weights, which results in a single network evaluation in seconds.
Our proposed method obtains a set of efficient models with state-of-the-art performance in two real-world search spaces.
arXiv Detail & Related papers (2021-10-08T06:35:20Z) - PEng4NN: An Accurate Performance Estimation Engine for Efficient
Automated Neural Network Architecture Search [0.0]
Neural network (NN) models are increasingly used in scientific simulations, AI, and other high performance computing fields.
NAS attempts to find well-performing NN models for specialized datsets, where performance is measured by key metrics that capture the NN capabilities.
We propose a performance estimation strategy that reduces the resources for training NNs and increases NAS throughput without jeopardizing accuracy.
arXiv Detail & Related papers (2021-01-11T20:49:55Z) - ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators
using Reinforcement Learning [5.251940442946459]
We propose an autonomous strategy called ConfuciuX to find optimized HW resource assignments for a given model and dataflow style.
It converges to the optimized hardware configuration 4.7 to 24 times faster than alternate techniques.
arXiv Detail & Related papers (2020-09-04T04:59:26Z) - FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining [65.39532971991778]
We present an accuracy predictor that scores architecture and training recipes jointly, guiding both sample selection and ranking.
We run fast evolutionary searches in just CPU minutes to generate architecture-recipe pairs for a variety of resource constraints.
FBNetV3 makes up a family of state-of-the-art compact neural networks that outperform both automatically and manually-designed competitors.
arXiv Detail & Related papers (2020-06-03T05:20:21Z) - DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search [76.9225014200746]
Efficient search is a core issue in Neural Architecture Search (NAS)
We present DA-NAS that can directly search the architecture for large-scale target tasks while allowing a large candidate set in a more efficient manner.
It is 2x faster than previous methods while the accuracy is currently state-of-the-art, at 76.2% under small FLOPs constraint.
arXiv Detail & Related papers (2020-03-27T17:55:21Z) - Widening and Squeezing: Towards Accurate and Efficient QNNs [125.172220129257]
Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters.
Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques.
We address this problem by projecting features in original full-precision networks to high-dimensional quantization features.
arXiv Detail & Related papers (2020-02-03T04:11:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.