Winograd Convolution: A Perspective from Fault Tolerance

• URL: http://arxiv.org/abs/2202.08675v1
• Date: Thu, 17 Feb 2022 14:19:55 GMT
• Title: Winograd Convolution: A Perspective from Fault Tolerance
• Authors: Xinghua Xue, Haitong Huang, Cheng Liu, Ying Wang, Tao Luo, Lei Zhang
• Abstract summary: Winograd convolution is originally proposed to reduce the computing overhead by converting multiplication in neural network (NN) with addition via linear transformation. We observe its great potential in improving NN fault tolerance and evaluate its fault tolerance for the first time.
• Score: 11.626169868836955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Winograd convolution is originally proposed to reduce the computing overhead by converting multiplication in neural network (NN) with addition via linear transformation. Other than the computing efficiency, we observe its great potential in improving NN fault tolerance and evaluate its fault tolerance comprehensively for the first time. Then, we explore the use of fault tolerance of winograd convolution for either fault-tolerant or energy-efficient NN processing. According to our experiments, winograd convolution can be utilized to reduce fault-tolerant design overhead by 27.49\% or energy consumption by 7.19\% without any accuracy loss compared to that without being aware of the fault tolerance

Related papers

• Towards Generalization in Subitizing with Neuro-Symbolic Loss using Holographic Reduced Representations [49.22640185566807]
  We show that adapting tools used in CogSci research can improve the subitizing generalization of CNNs and ViTs. We investigate how this neuro-symbolic approach to learning affects the subitizing capability of CNNs and ViTs. We find that ViTs perform considerably worse compared to CNNs in most respects on subitizing, except on one axis where an HRR-based loss provides improvement.
  arXiv  Detail & Related papers  (2023-12-23T17:54:03Z)
• Exploring Winograd Convolution for Cost-effective Neural Network Fault Tolerance [14.588891723027892]
  Winograd convolution can reduce the fault-tolerant design overhead by 55.77% on average without any accuracy loss compared to standard convolution. When it is applied on fault-tolerant neural networks enhanced with fault-aware retraining and constrained activation functions, the resulting model accuracy generally shows significant improvement in presence of various faults.
  arXiv  Detail & Related papers  (2023-08-16T09:03:13Z)
• Guaranteed Approximation Bounds for Mixed-Precision Neural Operators [83.64404557466528]
  We build on intuition that neural operator learning inherently induces an approximation error. We show that our approach reduces GPU memory usage by up to 50% and improves throughput by 58% with little or no reduction in accuracy.
  arXiv  Detail & Related papers  (2023-07-27T17:42:06Z)
• Low-Rank Winograd Transformation for 3D Convolutional Neural Networks [25.236436823266203]
  This paper focuses on Winograd transformation in 3D convolutional neural networks (CNNs) We introduce a low-rank Winograd transformation, a novel training paradigm that decouples the original large tensor into two less storage-required trainable tensors. We show that our proposed low-rank oriented sparse granularity permits practical Winograd acceleration compared with the vanilla counterpart.
  arXiv  Detail & Related papers  (2023-01-26T15:44:22Z)
• Winograd Algorithm for AdderNet [54.93995545896655]
  Adder neural network (AdderNet) is a new kind of deep model that replaces the original massive multiplications in convolutions by additions. This paper studies the winograd algorithm, which is a widely used fast algorithm for accelerating convolution and saving the computational costs.
  arXiv  Detail & Related papers  (2021-05-12T09:13:34Z)
• Efficient Residue Number System Based Winograd Convolution [15.210764522845416]
  Winograd algorithm can reduce the computational complexity of convolutional neural networks (CNN) with weights and activations represented in floating point. Our work extends the Winograd algorithm to Residue Number System (RNS) The minimal complexity convolution is computed precisely over large transformation tile.
  arXiv  Detail & Related papers  (2020-07-23T19:07:06Z)
• Rethinking Bottleneck Structure for Efficient Mobile Network Design [154.47657111869552]
  The inverted residual block is dominating architecture design for mobile networks recently. We propose to flip the structure and present a novel bottleneck design, called the sandglass block, that performs identity mapping and spatial transformation at higher dimensions. In ImageNet classification, by simply replacing the inverted residual block with our sandglass block without increasing parameters and computation, the classification accuracy can be improved by more than 1.7% over MobileNetV2.
  arXiv  Detail & Related papers  (2020-07-05T08:55:26Z)
• Quantaized Winograd/Toom-Cook Convolution for DNNs: Beyond Canonical Polynomials Base [0.0]
  Winograd convolution algorithm is a common used method that significantly reduces time consumption. We present the application of base change technique for quantized Winograd-aware training model.
  arXiv  Detail & Related papers  (2020-04-23T11:15:27Z)
• Detached Error Feedback for Distributed SGD with Random Sparsification [98.98236187442258]
  Communication bottleneck has been a critical problem in large-scale deep learning. We propose a new distributed error feedback (DEF) algorithm, which shows better convergence than error feedback for non-efficient distributed problems. We also propose DEFA to accelerate the generalization of DEF, which shows better bounds than DEF.
  arXiv  Detail & Related papers  (2020-04-11T03:50:59Z)
• LANCE: Efficient Low-Precision Quantized Winograd Convolution for Neural Networks Based on Graphics Processing Units [6.110973485878557]
  We propose an efficient low-precision quantized Winograd convolution algorithm, called LANCE, which combines the advantages of fast convolution and quantization techniques. We show that our 8-bit quantized Winograd convolution improves the performance by up to 2.40x over the full-precision convolution with trivial accuracy loss.
  arXiv  Detail & Related papers  (2020-03-19T09:46:50Z)
• Towards Unified INT8 Training for Convolutional Neural Network [83.15673050981624]
  We build a unified 8-bit (INT8) training framework for common convolutional neural networks. First, we empirically find the four distinctive characteristics of gradients, which provide us insightful clues for gradient quantization. We propose two universal techniques, including Direction Sensitive Gradient Clipping that reduces the direction deviation of gradients.
  arXiv  Detail & Related papers  (2019-12-29T08:37:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.