Improve Machine Learning carbon footprint using Nvidia GPU and Mixed Precision training for classification models -- Part I

• URL: http://arxiv.org/abs/2409.07853v2
• Date: Sat, 28 Sep 2024 20:38:17 GMT
• Title: Improve Machine Learning carbon footprint using Nvidia GPU and Mixed Precision training for classification models -- Part I
• Authors: Andrew Antonopoulos,
• Abstract summary: This dissertation compares the power consumption using the default floating point (32bit) and Nvidia mixed precision (16bit and 32bit) while training a classification ML model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This is the 1st part of the dissertation for my master degree and compares the power consumption using the default floating point (32bit) and Nvidia mixed precision (16bit and 32bit) while training a classification ML model. A custom PC with specific hardware was built to perform the experiments, and different ML hyper-parameters, such as batch size, neurons, and epochs, were chosen to build Deep Neural Networks (DNN). Additionally, various software was used during the experiments to collect the power consumption data in Watts from the Graphics Processing Unit (GPU), Central Processing Unit (CPU), Random Access Memory (RAM) and manually from a wattmeter connected to the wall. A benchmarking test with default hyper parameter values for the DNN was used as a reference, while the experiments used a combination of different settings. The results were recorded in Excel, and descriptive statistics were chosen to calculate the mean between the groups and compare them using graphs and tables. The outcome was positive when using mixed precision combined with specific hyper-parameters. Compared to the benchmarking, the optimisation for the classification reduced the power consumption between 7 and 11 Watts. Similarly, the carbon footprint is reduced because the calculation uses the same power consumption data. Still, a consideration is required when configuring hyper-parameters because it can negatively affect hardware performance. However, this research required inferential statistics, specifically ANOVA and T-test, to compare the relationship between the means. Furthermore, tests indicated no statistical significance of the relationship between the benchmarking and experiments. However, a more extensive implementation with a cluster of GPUs can increase the sample size significantly, as it is an essential factor and can change the outcome of the statistical analysis.

Related papers

• Improve Machine Learning carbon footprint using Parquet dataset format and Mixed Precision training for regression models -- Part II [0.0]
  This dissertation compared the power consumption using the Comma-Separated-Values (CSV) and dataset format with the default floating point (32bit) and Nvidia mixed precision (16bit and 32bit) while training a regression ML model.
  arXiv  Detail & Related papers  (2024-09-17T10:53:03Z)
• Be aware of overfitting by hyperparameter optimization! [0.0]
  We show that hyperparameter optimization did not always result in better models, possibly due to overfitting when using the same statistical measures. We also extended the previous analysis by adding a representation learning method based on Natural Language Processing of smiles called Transformer CNN. We show that across all analyzed sets using exactly the same protocol, Transformer CNN provided better results than graph-based methods for 26 out of 28 pairwise comparisons.
  arXiv  Detail & Related papers  (2024-07-30T12:45:05Z)
• Stabilizing Subject Transfer in EEG Classification with Divergence Estimation [17.924276728038304]
  We propose several graphical models to describe an EEG classification task. We identify statistical relationships that should hold true in an idealized training scenario. We design regularization penalties to enforce these relationships in two stages.
  arXiv  Detail & Related papers  (2023-10-12T23:06:52Z)
• Optimizing transformer-based machine translation model for single GPU training: a hyperparameter ablation study [0.0]
  In machine translation tasks, the relationship between model complexity and performance is often presumed to be linear. This study systematically investigates the effects of hyperparameters through ablation on a sequence-to-sequence machine translation pipeline. Contrary to expectations, our experiments reveal that combinations with the most parameters were not necessarily the most effective.
  arXiv  Detail & Related papers  (2023-08-11T08:47:52Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Machine Learning Capability: A standardized metric using case difficulty with applications to individualized deployment of supervised machine learning [2.2060666847121864]
  Model evaluation is a critical component in supervised machine learning classification analyses. Items Response Theory (IRT) and Computer Adaptive Testing (CAT) with machine learning can benchmark datasets independent of the end-classification results.
  arXiv  Detail & Related papers  (2023-02-09T00:38:42Z)
• Scaling Structured Inference with Randomization [64.18063627155128]
  We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states. Our method is widely applicable to classical DP-based inference. It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
  arXiv  Detail & Related papers  (2021-12-07T11:26:41Z)
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers [65.60007071024629]
  We show that Adaptive SGD outperforms four state-of-the-art solutions in time-to-accuracy. We show experimentally that Adaptive SGD outperforms four state-of-the-art solutions in time-to-accuracy.
  arXiv  Detail & Related papers  (2021-10-13T20:58:15Z)
• Statistical model-based evaluation of neural networks [74.10854783437351]
  We develop an experimental setup for the evaluation of neural networks (NNs) The setup helps to benchmark a set of NNs vis-a-vis minimum-mean-square-error (MMSE) performance bounds. This allows us to test the effects of training data size, data dimension, data geometry, noise, and mismatch between training and testing conditions.
  arXiv  Detail & Related papers  (2020-11-18T00:33:24Z)
• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers [86.36020260204302]
  We propose a new benchmarking protocol to evaluate both end-to-end efficiency and data-addition training efficiency. A human study is conducted to show that our evaluation protocol matches human tuning behavior better than the random search. We then apply the proposed benchmarking framework to 7s and various tasks, including computer vision, natural language processing, reinforcement learning, and graph mining.
  arXiv  Detail & Related papers  (2020-10-19T21:46:39Z)
• Rethinking the Hyperparameters for Fine-tuning [78.15505286781293]
  Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyper parameters. This paper re-examines several common practices of setting hyper parameters for fine-tuning.
  arXiv  Detail & Related papers  (2020-02-19T18:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.