Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities

• URL: http://arxiv.org/abs/2210.06640v2
• Date: Tue, 21 Mar 2023 08:24:01 GMT
• Title: Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities
• Authors: Brian R. Bartoldson, Bhavya Kailkhura, Davis Blalock
• Abstract summary: Economic and environmental costs of training neural networks are becoming unsustainable. Research on *algorithmically-efficient deep learning* seeks to reduce training costs through changes in the semantics of the training program. We formalize the *algorithmic speedup* problem, then use fundamental building blocks of algorithmically efficient training to develop a taxonomy.
• Score: 18.508401650991434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although deep learning has made great progress in recent years, the exploding economic and environmental costs of training neural networks are becoming unsustainable. To address this problem, there has been a great deal of research on *algorithmically-efficient deep learning*, which seeks to reduce training costs not at the hardware or implementation level, but through changes in the semantics of the training program. In this paper, we present a structured and comprehensive overview of the research in this field. First, we formalize the *algorithmic speedup* problem, then we use fundamental building blocks of algorithmically efficient training to develop a taxonomy. Our taxonomy highlights commonalities of seemingly disparate methods and reveals current research gaps. Next, we present evaluation best practices to enable comprehensive, fair, and reliable comparisons of speedup techniques. To further aid research and applications, we discuss common bottlenecks in the training pipeline (illustrated via experiments) and offer taxonomic mitigation strategies for them. Finally, we highlight some unsolved research challenges and present promising future directions.

Related papers

• Amortized nonmyopic active search via deep imitation learning [16.037812098340343]
  Active search formalizes a specialized active learning setting where the goal is to collect members of a rare, valuable class. We study the amortization of this policy by training a neural network to learn to search. Our network, trained on synthetic data, learns a beneficial search strategy that yields nonmyopic decisions.
  arXiv  Detail & Related papers  (2024-05-23T20:10:29Z)
• Neural Active Learning Beyond Bandits [69.99592173038903]
  We study both stream-based and pool-based active learning with neural network approximations. We propose two algorithms based on the newly designed exploitation and exploration neural networks for stream-based and pool-based active learning.
  arXiv  Detail & Related papers  (2024-04-18T21:52:14Z)
• Computation-efficient Deep Learning for Computer Vision: A Survey [121.84121397440337]
  Deep learning models have reached or even exceeded human-level performance in a range of visual perception tasks. Deep learning models usually demand significant computational resources, leading to impractical power consumption, latency, or carbon emissions in real-world scenarios. New research focus is computationally efficient deep learning, which strives to achieve satisfactory performance while minimizing the computational cost during inference.
  arXiv  Detail & Related papers  (2023-08-27T03:55:28Z)
• Actively Learning Costly Reward Functions for Reinforcement Learning [56.34005280792013]
  We show that it is possible to train agents in complex real-world environments orders of magnitudes faster. By enabling the application of reinforcement learning methods to new domains, we show that we can find interesting and non-trivial solutions.
  arXiv  Detail & Related papers  (2022-11-23T19:17:20Z)
• Pretraining in Deep Reinforcement Learning: A Survey [17.38360092869849]
  Pretraining has shown to be effective in acquiring transferable knowledge. Due to the nature of reinforcement learning, pretraining in this field is faced with unique challenges.
  arXiv  Detail & Related papers  (2022-11-08T02:17:54Z)
• Research Trends and Applications of Data Augmentation Algorithms [77.34726150561087]
  We identify the main areas of application of data augmentation algorithms, the types of algorithms used, significant research trends, their progression over time and research gaps in data augmentation literature. We expect readers to understand the potential of data augmentation, as well as identify future research directions and open questions within data augmentation research.
  arXiv  Detail & Related papers  (2022-07-18T11:38:32Z)
• Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms [1.2183405753834562]
  Backpropagation is the default algorithm for training deep neural networks due to its simplicity, efficiency and high convergence rate. In recent years, more biologically plausible learning methods have been proposed. BioTorch is a software framework to create, train, and benchmark biologically motivated neural networks.
  arXiv  Detail & Related papers  (2021-08-30T18:02:55Z)
• Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms [91.3755431537592]
  We analyze four broad meta-learning strategies which rely on plug-in estimation and pseudo-outcome regression. We highlight how this theoretical reasoning can be used to guide principled algorithm design and translate our analyses into practice.
  arXiv  Detail & Related papers  (2021-01-26T17:11:40Z)
• A Survey of Deep Meta-Learning [1.2891210250935143]
  Deep neural networks can achieve great successes when presented with large data sets and sufficient computational resources. However, their ability to learn new concepts quickly is limited. Deep Meta-Learning is one approach to address this issue, by enabling the network to learn how to learn.
  arXiv  Detail & Related papers  (2020-10-07T17:09:02Z)
• Binary Neural Networks: A Survey [126.67799882857656]
  The binary neural network serves as a promising technique for deploying deep models on resource-limited devices. The binarization inevitably causes severe information loss, and even worse, its discontinuity brings difficulty to the optimization of the deep network. We present a survey of these algorithms, mainly categorized into the native solutions directly conducting binarization, and the optimized ones using techniques like minimizing the quantization error, improving the network loss function, and reducing the gradient error.
  arXiv  Detail & Related papers  (2020-03-31T16:47:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.