How to use model architecture and training environment to estimate the energy consumption of DL training

• URL: http://arxiv.org/abs/2307.05520v4
• Date: Thu, 21 Nov 2024 19:09:26 GMT
• Title: How to use model architecture and training environment to estimate the energy consumption of DL training
• Authors: Santiago del Rey, Silverio Martínez-Fernández, Luís Cruz, Xavier Franch,
• Abstract summary: This study aims to leverage the relationship between energy consumption and two relevant design decisions in Deep Learning training. We study the training's power consumption behavior and propose four new energy estimation methods. Our results show that selecting the proper model architecture and training environment can reduce energy consumption dramatically.
• Score: 5.190998244098203
• License:
• Abstract: To raise awareness of the huge impact Deep Learning (DL) has on the environment, several works have tried to estimate the energy consumption and carbon footprint of DL-based systems across their life cycle. However, the estimations for energy consumption in the training stage usually rely on assumptions that have not been thoroughly tested. This study aims to move past these assumptions by leveraging the relationship between energy consumption and two relevant design decisions in DL training; model architecture, and training environment. To investigate these relationships, we collect multiple metrics related to energy efficiency and model correctness during the models' training. Then, we outline the trade-offs between the measured energy consumption and the models' correctness regarding model architecture, and their relationship with the training environment. Finally, we study the training's power consumption behavior and propose four new energy estimation methods. Our results show that selecting the proper model architecture and training environment can reduce energy consumption dramatically (up to 80.72%) at the cost of negligible decreases in correctness. Also, we find evidence that GPUs should scale with the models' computational complexity for better energy efficiency. Furthermore, we prove that current energy estimation methods are unreliable and propose alternatives 2x more precise.

Related papers

• Impact of ML Optimization Tactics on Greener Pre-Trained ML Models [46.78148962732881]
  This study aims to (i) analyze image classification datasets and pre-trained models, (ii) improve inference efficiency by comparing optimized and non-optimized models, and (iii) assess the economic impact of the optimizations. We conduct a controlled experiment to evaluate the impact of various PyTorch optimization techniques (dynamic quantization, torch.compile, local pruning, and global pruning) to 42 Hugging Face models for image classification. Dynamic quantization demonstrates significant reductions in inference time and energy consumption, making it highly suitable for large-scale systems.
  arXiv  Detail & Related papers  (2024-09-19T16:23:03Z)
• Computing Within Limits: An Empirical Study of Energy Consumption in ML Training and Inference [2.553456266022126]
  Machine learning (ML) has seen tremendous advancements, but its environmental footprint remains a concern. Acknowledging the growing environmental impact of ML this paper investigates Green ML.
  arXiv  Detail & Related papers  (2024-06-20T13:59:34Z)
• Watt For What: Rethinking Deep Learning's Energy-Performance Relationship [13.505163099299025]
  We study the trade-off between model accuracy and electricity consumption of deep learning models. By evaluating accuracy per unit of electricity consumed, we demonstrate how smaller, more energy-efficient models can significantly expedite research. This research contributes to a more equitable research landscape, where smaller entities can compete effectively with larger counterparts.
  arXiv  Detail & Related papers  (2023-10-10T11:08:31Z)
• Scaling Laws for Sparsely-Connected Foundation Models [70.41266138010657]
  We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets. We identify the first scaling law describing the relationship between weight sparsity, number of non-zero parameters, and amount of training data.
  arXiv  Detail & Related papers  (2023-09-15T16:29:27Z)
• A Comparative Study of Machine Learning Algorithms for Anomaly Detection in Industrial Environments: Performance and Environmental Impact [62.997667081978825]
  This study seeks to address the demands of high-performance machine learning models with environmental sustainability. Traditional machine learning algorithms, such as Decision Trees and Random Forests, demonstrate robust efficiency and performance. However, superior outcomes were obtained with optimised configurations, albeit with a commensurate increase in resource consumption.
  arXiv  Detail & Related papers  (2023-07-01T15:18:00Z)
• Accuracy is not the only Metric that matters: Estimating the Energy Consumption of Deep Learning Models [33.45069308137142]
  We have created an energy estimation pipeline1, which allows practitioners to estimate the energy needs of their models in advance, without actually running or training them. We accomplished this, by collecting high-quality energy data and building a first baseline model, capable of predicting the energy consumption of DL models by accumulating their estimated layer-wise energies.
  arXiv  Detail & Related papers  (2023-04-03T11:35:10Z)
• Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learning [77.62876532784759]
  Machine learning (ML) requires using energy to carry out computations during the model training process. The generation of this energy comes with an environmental cost in terms of greenhouse gas emissions, depending on quantity used and the energy source. We present a survey of the carbon emissions of 95 ML models across time and different tasks in natural language processing and computer vision.
  arXiv  Detail & Related papers  (2023-02-16T18:35:00Z)
• Energy Efficiency of Training Neural Network Architectures: An Empirical Study [11.325530936177493]
  The evaluation of Deep Learning models has traditionally focused on criteria such as accuracy, F1 score, and related measures. The computations needed to train such models entail a large carbon footprint. We study the relations between DL model architectures and their environmental impact in terms of energy consumed and CO$$ emissions produced during training.
  arXiv  Detail & Related papers  (2023-02-02T09:20:54Z)
• Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision [31.781943982148025]
  We present the first large-scale energy consumption benchmark for efficient computer vision models. A new metric is proposed to explicitly evaluate the full-cycle energy consumption under different model usage intensity.
  arXiv  Detail & Related papers  (2021-08-30T18:22:36Z)
• HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing [76.38975568873765]
  We introduce HULK, a multi-task energy efficiency benchmarking platform for responsible natural language processing. We compare pretrained models' energy efficiency from the perspectives of time and cost.
  arXiv  Detail & Related papers  (2020-02-14T01:04:19Z)
• NeurOpt: Neural network based optimization for building energy management and climate control [58.06411999767069]
  We propose a data-driven control algorithm based on neural networks to reduce this cost of model identification. We validate our learning and control algorithms on a two-story building with ten independently controlled zones, located in Italy.
  arXiv  Detail & Related papers  (2020-01-22T00:51:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.