Carbon Emissions and Large Neural Network Training

• URL: http://arxiv.org/abs/2104.10350v3
• Date: Fri, 23 Apr 2021 14:26:29 GMT
• Title: Carbon Emissions and Large Neural Network Training
• Authors: David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, Jeff Dean
• Abstract summary: We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e) To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models.
• Score: 19.233899715628073
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information. We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary ~5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be ~2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X. These large factors also make retroactive estimates of energy cost difficult. To avoid miscalculations, we believe ML papers requiring large computational resources should make energy consumption and CO2e explicit when practical. We are working to be more transparent about energy use and CO2e in our future research. To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.

Related papers

• Generative AI for Low-Carbon Artificial Intelligence of Things with Large Language Models [67.0243099823109]
  Generative AI (GAI) holds immense potential to reduce carbon emissions of Artificial Intelligence of Things (AIoT) In this article, we explore the potential of GAI for carbon emissions reduction and propose a novel GAI-enabled solution for low-carbon AIoT. We propose a Large Language Model (LLM)-enabled carbon emission optimization framework, in which we design pluggable LLM and Retrieval Augmented Generation (RAG) modules.
  arXiv  Detail & Related papers  (2024-04-28T05:46:28Z)
• Photonics for Sustainable Computing [1.7396686601746498]
  Photonic integrated circuits are finding use in a variety of applications including optical transceivers, LIDAR, bio-sensing, photonic quantum computing, and Machine Learning. In this paper, we build a carbon footprint model for photonic chips and investigate the sustainability of photonics-based accelerators. Our analysis shows that photonics can reduce both operational and embodied carbon footprints with its high energy efficiency.
  arXiv  Detail & Related papers  (2024-01-10T12:37:23Z)
• Green Federated Learning [7.003870178055125]
  Federated Learning (FL) is a machine learning technique for training a centralized model using data of decentralized entities. FL may leverage as many as hundreds of millions of globally distributed end-user devices with diverse energy sources. We propose the concept of Green FL, which involves optimizing FL parameters and making design choices to minimize carbon emissions.
  arXiv  Detail & Related papers  (2023-03-26T02:23:38Z)
• Counting Carbon: A Survey of Factors Influencing the Emissions of Machine Learning [77.62876532784759]
  Machine learning (ML) requires using energy to carry out computations during the model training process. The generation of this energy comes with an environmental cost in terms of greenhouse gas emissions, depending on quantity used and the energy source. We present a survey of the carbon emissions of 95 ML models across time and different tasks in natural language processing and computer vision.
  arXiv  Detail & Related papers  (2023-02-16T18:35:00Z)
• PhAST: Physics-Aware, Scalable, and Task-specific GNNs for Accelerated Catalyst Design [102.9593507372373]
  Catalyst materials play a crucial role in the electrochemical reactions involved in industrial processes. Machine learning holds the potential to efficiently model materials properties from large amounts of data. We propose task-specific innovations applicable to most architectures, enhancing both computational efficiency and accuracy.
  arXiv  Detail & Related papers  (2022-11-22T05:24:30Z)
• Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model [72.65502770895417]
  We quantify the carbon footprint of BLOOM, a 176-billion parameter language model, across its life cycle. We estimate that BLOOM's final training emitted approximately 24.7 tonnes ofcarboneqif we consider only the dynamic power consumption. We conclude with a discussion regarding the difficulty of precisely estimating the carbon footprint of machine learning models.
  arXiv  Detail & Related papers  (2022-11-03T17:13:48Z)
• Measuring the Carbon Intensity of AI in Cloud Instances [91.28501520271972]
  We provide a framework for measuring software carbon intensity, and propose to measure operational carbon emissions. We evaluate a suite of approaches for reducing emissions on the Microsoft Azure cloud compute platform.
  arXiv  Detail & Related papers  (2022-06-10T17:04:04Z)
• The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink [14.427445867512366]
  Four best practices can reduce ML training energy by up to 100x and CO2 emissions up to 1000x. By following best practices, overall ML energy use held steady at 15% of Google's total energy use for the past three years. We recommend that ML papers include emissions explicitly to foster competition on more than just model quality.
  arXiv  Detail & Related papers  (2022-04-11T14:30:27Z)
• Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning [68.37641996188133]
  We introduce a framework for tracking realtime energy consumption and carbon emissions. We create a leaderboard for energy efficient reinforcement learning algorithms. We propose strategies for mitigation of carbon emissions and reduction of energy consumption.
  arXiv  Detail & Related papers  (2020-01-31T05:12:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.