The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

• URL: http://arxiv.org/abs/2204.05149v1
• Date: Mon, 11 Apr 2022 14:30:27 GMT
• Title: The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink
• Authors: David Patterson, Joseph Gonzalez, Urs H\"olzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean
• Abstract summary: Four best practices can reduce ML training energy by up to 100x and CO2 emissions up to 1000x. By following best practices, overall ML energy use held steady at 15% of Google's total energy use for the past three years. We recommend that ML papers include emissions explicitly to foster competition on more than just model quality.
• Score: 14.427445867512366
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine Learning (ML) workloads have rapidly grown in importance, but raised concerns about their carbon footprint. Four best practices can reduce ML training energy by up to 100x and CO2 emissions up to 1000x. By following best practices, overall ML energy use (across research, development, and production) held steady at <15% of Google's total energy use for the past three years. If the whole ML field were to adopt best practices, total carbon emissions from training would reduce. Hence, we recommend that ML papers include emissions explicitly to foster competition on more than just model quality. Estimates of emissions in papers that omitted them have been off 100x-100,000x, so publishing emissions has the added benefit of ensuring accurate accounting. Given the importance of climate change, we must get the numbers right to make certain that we work on its biggest challenges.

Related papers

• Reporting and Analysing the Environmental Impact of Language Models on the Example of Commonsense Question Answering with External Knowledge [7.419725234099729]
  ChatGPT sparked social interest in Large Language Models (LLMs) LLMs demand substantial computational resources and are very costly to train, both financially and environmentally. In this study, we infused T5 LLM with external knowledge and fine-tuned the model for Question-Answering task.
  arXiv  Detail & Related papers  (2024-07-24T16:16:16Z)
• Generative AI for Low-Carbon Artificial Intelligence of Things with Large Language Models [67.0243099823109]
  Generative AI (GAI) holds immense potential to reduce carbon emissions of Artificial Intelligence of Things (AIoT) In this article, we explore the potential of GAI for carbon emissions reduction and propose a novel GAI-enabled solution for low-carbon AIoT. We propose a Large Language Model (LLM)-enabled carbon emission optimization framework, in which we design pluggable LLM and Retrieval Augmented Generation (RAG) modules.
  arXiv  Detail & Related papers  (2024-04-28T05:46:28Z)
• Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learning [77.62876532784759]
  Machine learning (ML) requires using energy to carry out computations during the model training process. The generation of this energy comes with an environmental cost in terms of greenhouse gas emissions, depending on quantity used and the energy source. We present a survey of the carbon emissions of 95 ML models across time and different tasks in natural language processing and computer vision.
  arXiv  Detail & Related papers  (2023-02-16T18:35:00Z)
• PhAST: Physics-Aware, Scalable, and Task-specific GNNs for Accelerated Catalyst Design [102.9593507372373]
  Catalyst materials play a crucial role in the electrochemical reactions involved in industrial processes. Machine learning holds the potential to efficiently model materials properties from large amounts of data. We propose task-specific innovations applicable to most architectures, enhancing both computational efficiency and accuracy.
  arXiv  Detail & Related papers  (2022-11-22T05:24:30Z)
• Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model [72.65502770895417]
  We quantify the carbon footprint of BLOOM, a 176-billion parameter language model, across its life cycle. We estimate that BLOOM's final training emitted approximately 24.7 tonnes ofcarboneqif we consider only the dynamic power consumption. We conclude with a discussion regarding the difficulty of precisely estimating the carbon footprint of machine learning models.
  arXiv  Detail & Related papers  (2022-11-03T17:13:48Z)
• Measuring the Carbon Intensity of AI in Cloud Instances [91.28501520271972]
  We provide a framework for measuring software carbon intensity, and propose to measure operational carbon emissions. We evaluate a suite of approaches for reducing emissions on the Microsoft Azure cloud compute platform.
  arXiv  Detail & Related papers  (2022-06-10T17:04:04Z)
• Carbon Emissions and Large Neural Network Training [19.233899715628073]
  We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e) To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models.
  arXiv  Detail & Related papers  (2021-04-21T04:44:25Z)
• Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models [0.3441021278275805]
  Machine learning (ML) may become a significant contributor to climate change if this exponential trend continues. We propose that energy and carbon footprint of model development and training is reported alongside performance metrics using tools like Carbontracker.
  arXiv  Detail & Related papers  (2020-07-06T20:24:31Z)
• Optimizing carbon tax for decentralized electricity markets using an agent-based model [69.3939291118954]
  Averting the effects of anthropogenic climate change requires a transition from fossil fuels to low-carbon technology. Carbon taxes have been shown to be an efficient way to aid in this transition. We use the NSGA-II genetic algorithm to minimize average electricity price and relative carbon intensity of the electricity mix.
  arXiv  Detail & Related papers  (2020-05-28T06:54:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.