A Survey on Design Methodologies for Accelerating Deep Learning on Heterogeneous Architectures

• URL: http://arxiv.org/abs/2311.17815v1
• Date: Wed, 29 Nov 2023 17:10:16 GMT
• Title: A Survey on Design Methodologies for Accelerating Deep Learning on Heterogeneous Architectures
• Authors: Fabrizio Ferrandi, Serena Curzel, Leandro Fiorin, Daniele Ielmini, Cristina Silvano, Francesco Conti, Alessio Burrello, Francesco Barchi, Luca Benini, Luciano Lavagno, Teodoro Urso, Enrico Calore, Sebastiano Fabio Schifano, Cristian Zambelli, Maurizio Palesi, Giuseppe Ascia, Enrico Russo, Nicola Petra, Davide De Caro, Gennaro Di Meo, Valeria Cardellini, Salvatore Filippone, Francesco Lo Presti, Francesco Silvestri, Paolo Palazzari and Stefania Perri
• Abstract summary: The need for efficient hardware accelerators has become more pressing to design heterogeneous HPC platforms. Several methodologies and tools have been proposed to design accelerators for Deep Learning. This survey provides a holistic review of the most influential design methodologies and EDA tools proposed in recent years to implement Deep Learning accelerators.
• Score: 9.982620766142345
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In recent years, the field of Deep Learning has seen many disruptive and impactful advancements. Given the increasing complexity of deep neural networks, the need for efficient hardware accelerators has become more and more pressing to design heterogeneous HPC platforms. The design of Deep Learning accelerators requires a multidisciplinary approach, combining expertise from several areas, spanning from computer architecture to approximate computing, computational models, and machine learning algorithms. Several methodologies and tools have been proposed to design accelerators for Deep Learning, including hardware-software co-design approaches, high-level synthesis methods, specific customized compilers, and methodologies for design space exploration, modeling, and simulation. These methodologies aim to maximize the exploitable parallelism and minimize data movement to achieve high performance and energy efficiency. This survey provides a holistic review of the most influential design methodologies and EDA tools proposed in recent years to implement Deep Learning accelerators, offering the reader a wide perspective in this rapidly evolving field. In particular, this work complements the previous survey proposed by the same authors in [203], which focuses on Deep Learning hardware accelerators for heterogeneous HPC platforms.

Related papers

• Inference Optimization of Foundation Models on AI Accelerators [68.24450520773688]
  Powerful foundation models, including large language models (LLMs), with Transformer architectures have ushered in a new era of Generative AI. As the number of model parameters reaches to hundreds of billions, their deployment incurs prohibitive inference costs and high latency in real-world scenarios. This tutorial offers a comprehensive discussion on complementary inference optimization techniques using AI accelerators.
  arXiv  Detail & Related papers  (2024-07-12T09:24:34Z)
• Design Space Exploration of Approximate Computing Techniques with a Reinforcement Learning Approach [49.42371633618761]
  We propose an RL-based strategy to find approximate versions of an application that balance accuracy degradation and power and computation time reduction. Our experimental results show a good trade-off between accuracy degradation and decreased power and computation time for some benchmarks.
  arXiv  Detail & Related papers  (2023-12-29T09:10:40Z)
• Computation-efficient Deep Learning for Computer Vision: A Survey [121.84121397440337]
  Deep learning models have reached or even exceeded human-level performance in a range of visual perception tasks. Deep learning models usually demand significant computational resources, leading to impractical power consumption, latency, or carbon emissions in real-world scenarios. New research focus is computationally efficient deep learning, which strives to achieve satisfactory performance while minimizing the computational cost during inference.
  arXiv  Detail & Related papers  (2023-08-27T03:55:28Z)
• Approximate Computing Survey, Part II: Application-Specific & Architectural Approximation Techniques and Applications [14.450131342802631]
  Approximate Computing allows to tune the quality of results in the design of a system in order to improve the energy efficiency and/or performance. This radical paradigm shift has attracted interest from both academia and industry. We conduct a two-part survey to cover key aspects (e.g., terminology and applications) and review the state-of-the art approximation techniques.
  arXiv  Detail & Related papers  (2023-07-20T15:54:33Z)
• A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms [9.036774656254375]
  This survey summarizes and classifies the most recent advances in designing deep learning accelerators. It highlights the most advanced approaches to support deep learning accelerations including not only GPU and TPU-based accelerators but also design-specific hardware accelerators. The survey also describes accelerators based on emerging memory technologies and computing paradigms.
  arXiv  Detail & Related papers  (2023-06-27T15:24:24Z)
• On Efficient Training of Large-Scale Deep Learning Models: A Literature Review [90.87691246153612]
  The field of deep learning has witnessed significant progress, particularly in computer vision (CV), natural language processing (NLP), and speech. The use of large-scale models trained on vast amounts of data holds immense promise for practical applications. With the increasing demands on computational capacity, a comprehensive summarization on acceleration techniques of training deep learning models is still much anticipated.
  arXiv  Detail & Related papers  (2023-04-07T11:13:23Z)
• Hybrid Supervised and Reinforcement Learning for the Design and Optimization of Nanophotonic Structures [8.677532138573984]
  This paper presents a hybrid supervised and reinforcement learning approach to the inverse design of nanophotonic structures. We show this approach can reduce training data dependence, improve the generalizability of model predictions, and shorten exploratory training times by orders of magnitude.
  arXiv  Detail & Related papers  (2022-09-08T22:43:40Z)
• Dynamically Grown Generative Adversarial Networks [111.43128389995341]
  We propose a method to dynamically grow a GAN during training, optimizing the network architecture and its parameters together with automation. The method embeds architecture search techniques as an interleaving step with gradient-based training to periodically seek the optimal architecture-growing strategy for the generator and discriminator.
  arXiv  Detail & Related papers  (2021-06-16T01:25:51Z)
• Integrating Deep Learning in Domain Sciences at Exascale [2.241545093375334]
  We evaluate existing packages for their ability to run deep learning models and applications on large-scale HPC systems efficiently. We propose new asynchronous parallelization and optimization techniques for current large-scale heterogeneous systems. We present illustrations and potential solutions for enhancing traditional compute- and data-intensive applications with AI.
  arXiv  Detail & Related papers  (2020-11-23T03:09:58Z)
• Knowledge Distillation: A Survey [87.51063304509067]
  Deep neural networks have been successful in both industry and academia, especially for computer vision tasks. It is a challenge to deploy these cumbersome deep models on devices with limited resources. Knowledge distillation effectively learns a small student model from a large teacher model.
  arXiv  Detail & Related papers  (2020-06-09T21:47:17Z)
• Deep Learning and Knowledge-Based Methods for Computer Aided Molecular Design -- Toward a Unified Approach: State-of-the-Art and Future Directions [0.0]
  The optimal design of compounds through manipulating properties at the molecular level is often the key to scientific advances and improved process systems performance. This paper highlights key trends, challenges, and opportunities underpinning the Computer-Aided Molecular Design problems.
  arXiv  Detail & Related papers  (2020-05-18T14:17:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.