Third ArchEdge Workshop: Exploring the Design Space of Efficient Deep Neural Networks

• URL: http://arxiv.org/abs/2011.10912v1
• Date: Sun, 22 Nov 2020 01:56:46 GMT
• Title: Third ArchEdge Workshop: Exploring the Design Space of Efficient Deep Neural Networks
• Authors: Fuxun Yu, Dimitrios Stamoulis, Di Wang, Dimitrios Lymberopoulos, Xiang Chen
• Abstract summary: This paper gives an overview of our ongoing work on the design space exploration of efficient deep neural networks (DNNs) We cover two aspects: (1) static architecture design efficiency and (2) dynamic model execution efficiency. We highlight several open questions that are poised to draw research attention in the next few years.
• Score: 14.195694804273801
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper gives an overview of our ongoing work on the design space exploration of efficient deep neural networks (DNNs). Specifically, we cover two aspects: (1) static architecture design efficiency and (2) dynamic model execution efficiency. For static architecture design, different from existing end-to-end hardware modeling assumptions, we conduct full-stack profiling at the GPU core level to identify better accuracy-latency trade-offs for DNN designs. For dynamic model execution, different from prior work that tackles model redundancy at the DNN-channels level, we explore a new dimension of DNN feature map redundancy to be dynamically traversed at runtime. Last, we highlight several open questions that are poised to draw research attention in the next few years.

Related papers

• Towards Ideal Temporal Graph Neural Networks: Evaluations and Conclusions after 10,000 GPU Hours [4.536379628671518]
  Temporal Graph Neural Networks (TGNNs) have emerged as powerful tools for modeling dynamic interactions across various domains. We contend that many of the existing works on TGNN modeling inadequately explore the design space, leading to suboptimal designs. We propose a practical comparative evaluation framework that performs a design space search across well-known TGNN modules.
  arXiv  Detail & Related papers  (2024-12-28T20:02:29Z)
• Improvement of Spiking Neural Network with Bit Planes and Color Models [0.0]
  Spiking neural network (SNN) has emerged as a promising paradigm in computational neuroscience and artificial intelligence. We present a novel approach to enhance the performance of SNN for images through a new coding method that exploits bit plane representation.
  arXiv  Detail & Related papers  (2024-09-28T15:52:49Z)
• NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals [58.83169560132308]
  We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of very large neural networks. NNsight is an open-source system that extends PyTorch to introduce deferred remote execution. NDIF is a scalable inference service that executes NNsight requests, allowing users to share GPU resources and pretrained models.
  arXiv  Detail & Related papers  (2024-07-18T17:59:01Z)
• From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport [32.39176908225668]
  We introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for measuring the non-linearity of deep neural networks. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature.
  arXiv  Detail & Related papers  (2023-10-17T17:50:22Z)
• Flexible Channel Dimensions for Differentiable Architecture Search [50.33956216274694]
  We propose a novel differentiable neural architecture search method with an efficient dynamic channel allocation algorithm. We show that the proposed framework is able to find DNN architectures that are equivalent to previous methods in task accuracy and inference latency.
  arXiv  Detail & Related papers  (2023-06-13T15:21:38Z)
• Comparison Analysis of Traditional Machine Learning and Deep Learning Techniques for Data and Image Classification [62.997667081978825]
  The purpose of the study is to analyse and compare the most common machine learning and deep learning techniques used for computer vision 2D object classification tasks. Firstly, we will present the theoretical background of the Bag of Visual words model and Deep Convolutional Neural Networks (DCNN) Secondly, we will implement a Bag of Visual Words model, the VGG16 CNN Architecture.
  arXiv  Detail & Related papers  (2022-04-11T11:34:43Z)
• QuadraLib: A Performant Quadratic Neural Network Library for Architecture Optimization and Design Exploration [31.488940932186246]
  Quadratic Deep Neuron Networks (QDNNs) show better non-linearity and learning capability than the first-order DNNs. Our design has good performance regarding prediction accuracy and computation consumption on multiple learning tasks.
  arXiv  Detail & Related papers  (2022-04-01T18:06:54Z)
• Sparse Flows: Pruning Continuous-depth Models [107.98191032466544]
  We show that pruning improves generalization for neural ODEs in generative modeling. We also show that pruning finds minimal and efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy.
  arXiv  Detail & Related papers  (2021-06-24T01:40:17Z)
• A Design Space Study for LISTA and Beyond [79.76740811464597]
  In recent years, great success has been witnessed in building problem-specific deep networks from unrolling iterative algorithms. This paper revisits the role of unrolling as a design approach for deep networks, to what extent its resulting special architecture is superior, and can we find better? Using LISTA for sparse recovery as a representative example, we conduct the first thorough design space study for the unrolled models.
  arXiv  Detail & Related papers  (2021-04-08T23:01:52Z)
• Design Space for Graph Neural Networks [81.88707703106232]
  We study the architectural design space for Graph Neural Networks (GNNs) which consists of 315,000 different designs over 32 different predictive tasks. Our key results include: (1) A comprehensive set of guidelines for designing well-performing GNNs; (2) while best GNN designs for different tasks vary significantly, the GNN task space allows for transferring the best designs across different tasks; (3) models discovered using our design space achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-11-17T18:59:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.