Low-Rank Winograd Transformation for 3D Convolutional Neural Networks

• URL: http://arxiv.org/abs/2301.11180v1
• Date: Thu, 26 Jan 2023 15:44:22 GMT
• Title: Low-Rank Winograd Transformation for 3D Convolutional Neural Networks
• Authors: Ziran Qin, Mingbao Lin, Weiyao Lin
• Abstract summary: This paper focuses on Winograd transformation in 3D convolutional neural networks (CNNs) We introduce a low-rank Winograd transformation, a novel training paradigm that decouples the original large tensor into two less storage-required trainable tensors. We show that our proposed low-rank oriented sparse granularity permits practical Winograd acceleration compared with the vanilla counterpart.
• Score: 25.236436823266203
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper focuses on Winograd transformation in 3D convolutional neural networks (CNNs) that are more over-parameterized compared with the 2D version. The over-increasing Winograd parameters not only exacerbate training complexity but also barricade the practical speedups due simply to the volume of element-wise products in the Winograd domain. We attempt to reduce trainable parameters by introducing a low-rank Winograd transformation, a novel training paradigm that decouples the original large tensor into two less storage-required trainable tensors, leading to a significant complexity reduction. Built upon our low-rank Winograd transformation, we take one step ahead by proposing a low-rank oriented sparse granularity that measures column-wise parameter importance. By simply involving the non-zero columns in the element-wise product, our sparse granularity is empowered with the ability to produce a very regular sparse pattern to acquire effectual Winograd speedups. To better understand the efficacy of our method, we perform extensive experiments on 3D CNNs. Results manifest that our low-rank Winograd transformation well outperforms the vanilla Winograd transformation. We also show that our proposed low-rank oriented sparse granularity permits practical Winograd acceleration compared with the vanilla counterpart.

Related papers

• Data-Free Group-Wise Fully Quantized Winograd Convolution via Learnable Scales [4.1966303054440655]
  Quantization of diffusion models has been explored in recent works to reduce compute costs and memory bandwidth usage. For text-to-image generation task, the 8-bit fully-quantized diffusion model with Winograd provides near-lossless quality. For image classification, our method outperforms the state-of-the-art Winograd PTQ method by 1.62% and 2.56% in top-1 ImageNet accuracy.
  arXiv  Detail & Related papers  (2024-12-27T09:05:48Z)
• Exploring Winograd Convolution for Cost-effective Neural Network Fault Tolerance [14.588891723027892]
  Winograd convolution can reduce the fault-tolerant design overhead by 55.77% on average without any accuracy loss compared to standard convolution. When it is applied on fault-tolerant neural networks enhanced with fault-aware retraining and constrained activation functions, the resulting model accuracy generally shows significant improvement in presence of various faults.
  arXiv  Detail & Related papers  (2023-08-16T09:03:13Z)
• Fourier-Net+: Leveraging Band-Limited Representation for Efficient 3D Medical Image Registration [62.53130123397081]
  U-Net style networks are commonly utilized in unsupervised image registration to predict dense displacement fields. We first propose Fourier-Net, which replaces the costly U-Net style expansive path with a parameter-free model-driven decoder. We then introduce Fourier-Net+, which additionally takes the band-limited spatial representation of the images as input and further reduces the number of convolutional layers in the U-Net style network's contracting path.
  arXiv  Detail & Related papers  (2023-07-06T13:57:12Z)
• Rethinking Hierarchicies in Pre-trained Plain Vision Transformer [76.35955924137986]
  Self-supervised pre-training vision transformer (ViT) via masked image modeling (MIM) has been proven very effective. customized algorithms should be carefully designed for the hierarchical ViTs, e.g., GreenMIM, instead of using the vanilla and simple MAE for the plain ViT. This paper proposes a novel idea of disentangling the hierarchical architecture design from the self-supervised pre-training.
  arXiv  Detail & Related papers  (2022-11-03T13:19:23Z)
• GradViT: Gradient Inversion of Vision Transformers [83.54779732309653]
  We demonstrate the vulnerability of vision transformers (ViTs) to gradient-based inversion attacks. We introduce a method, named GradViT, that optimize random noise into naturally looking images. We observe unprecedentedly high fidelity and closeness to the original (hidden) data.
  arXiv  Detail & Related papers  (2022-03-22T17:06:07Z)
• Orthogonal Graph Neural Networks [53.466187667936026]
  Graph neural networks (GNNs) have received tremendous attention due to their superiority in learning node representations. stacking more convolutional layers significantly decreases the performance of GNNs. We propose a novel Ortho-GConv, which could generally augment the existing GNN backbones to stabilize the model training and improve the model's generalization performance.
  arXiv  Detail & Related papers  (2021-09-23T12:39:01Z)
• Winograd Algorithm for AdderNet [54.93995545896655]
  Adder neural network (AdderNet) is a new kind of deep model that replaces the original massive multiplications in convolutions by additions. This paper studies the winograd algorithm, which is a widely used fast algorithm for accelerating convolution and saving the computational costs.
  arXiv  Detail & Related papers  (2021-05-12T09:13:34Z)
• Accelerating Large Kernel Convolutions with Nested Winograd Transformation.pdf [2.193040410545991]
  This work proposes a nested Winograd algorithm that iteratively decomposes a large kernel convolution into small kernel convolutions. Experiments show that compared to the linear decomposition Winograd algorithm, the proposed algorithm reduces the total number of multiplications by 1.4 to 10.5 times for computing 4x4 to 31x31 convolutions.
  arXiv  Detail & Related papers  (2021-02-26T02:42:42Z)
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training [59.160154997555956]
  We present GradInit, an automated and architecture method for initializing neural networks. It is based on a simple agnostic; the variance of each network layer is adjusted so that a single step of SGD or Adam results in the smallest possible loss value. It also enables training the original Post-LN Transformer for machine translation without learning rate warmup.
  arXiv  Detail & Related papers  (2021-02-16T11:45:35Z)
• LANCE: Efficient Low-Precision Quantized Winograd Convolution for Neural Networks Based on Graphics Processing Units [6.110973485878557]
  We propose an efficient low-precision quantized Winograd convolution algorithm, called LANCE, which combines the advantages of fast convolution and quantization techniques. We show that our 8-bit quantized Winograd convolution improves the performance by up to 2.40x over the full-precision convolution with trivial accuracy loss.
  arXiv  Detail & Related papers  (2020-03-19T09:46:50Z)
• Searching for Winograd-aware Quantized Networks [12.351250944079949]
  We propose a Winograd-aware formulation of convolution layers which exposes the numerical inaccuracies introduced by the Winograd transformations. We also address the source of the numerical error and propose a relaxation on the form of the transformation matrices, resulting in up to 10% higher classification accuracy on CIFAR-10.
  arXiv  Detail & Related papers  (2020-02-25T07:53:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.