VQ4ALL: Efficient Neural Network Representation via a Universal Codebook

• URL: http://arxiv.org/abs/2412.06875v1
• Date: Mon, 09 Dec 2024 16:17:22 GMT
• Title: VQ4ALL: Efficient Neural Network Representation via a Universal Codebook
• Authors: Juncan Deng, Shuaiting Li, Zeyu Wang, Hong Gu, Kedong Xu, Kejie Huang,
• Abstract summary: We introduce VQ4ALL, a VQ-based method that utilizes codewords to enable the construction of various neural networks. VQ4ALL achieves compression rates exceeding 16 $times$ while preserving high accuracy across multiple network architectures.
• Score: 7.369445527610879
• License:
• Abstract: The rapid growth of the big neural network models puts forward new requirements for lightweight network representation methods. The traditional methods based on model compression have achieved great success, especially VQ technology which realizes the high compression ratio of models by sharing code words. However, because each layer of the network needs to build a code table, the traditional top-down compression technology lacks attention to the underlying commonalities, resulting in limited compression rate and frequent memory access. In this paper, we propose a bottom-up method to share the universal codebook among multiple neural networks, which not only effectively reduces the number of codebooks but also further reduces the memory access and chip area by storing static code tables in the built-in ROM. Specifically, we introduce VQ4ALL, a VQ-based method that utilizes codewords to enable the construction of various neural networks and achieve efficient representations. The core idea of our method is to adopt a kernel density estimation approach to extract a universal codebook and then progressively construct different low-bit networks by updating differentiable assignments. Experimental results demonstrate that VQ4ALL achieves compression rates exceeding 16 $\times$ while preserving high accuracy across multiple network architectures, highlighting its effectiveness and versatility.

Related papers

• Tiled Bit Networks: Sub-Bit Neural Network Compression Through Reuse of Learnable Binary Vectors [4.95475852994362]
  We propose a new form of quantization to tile neural network layers with sequences of bits to achieve sub-bit compression of binary-weighted neural networks. We employ the approach to both fully-connected and convolutional layers, which make up the breadth of space in most neural architectures.
  arXiv  Detail & Related papers  (2024-07-16T15:55:38Z)
• A Theoretical Understanding of Neural Network Compression from Sparse Linear Approximation [37.525277809849776]
  The goal of model compression is to reduce the size of a large neural network while retaining a comparable performance. We use sparsity-sensitive $ell_q$-norm to characterize compressibility and provide a relationship between soft sparsity of the weights in the network and the degree of compression. We also develop adaptive algorithms for pruning each neuron in the network informed by our theory.
  arXiv  Detail & Related papers  (2022-06-11T20:10:35Z)
• Compact representations of convolutional neural networks via weight pruning and quantization [63.417651529192014]
  We propose a novel storage format for convolutional neural networks (CNNs) based on source coding and leveraging both weight pruning and quantization. We achieve a reduction of space occupancy up to 0.6% on fully connected layers and 5.44% on the whole network, while performing at least as competitive as the baseline.
  arXiv  Detail & Related papers  (2021-08-28T20:39:54Z)
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration [83.84684675841167]
  We propose a novel encoding scheme using -1, +1 to decompose quantized neural networks (QNNs) into multi-branch binary networks. We validate the effectiveness of our method on large-scale image classification, object detection, and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-06-18T03:11:15Z)
• All at Once Network Quantization via Collaborative Knowledge Transfer [56.95849086170461]
  We develop a novel collaborative knowledge transfer approach for efficiently training the all-at-once quantization network. Specifically, we propose an adaptive selection strategy to choose a high-precision enquoteteacher for transferring knowledge to the low-precision student. To effectively transfer knowledge, we develop a dynamic block swapping method by randomly replacing the blocks in the lower-precision student network with the corresponding blocks in the higher-precision teacher network.
  arXiv  Detail & Related papers  (2021-03-02T03:09:03Z)
• Learned Multi-Resolution Variable-Rate Image Compression with Octave-based Residual Blocks [15.308823742699039]
  We propose a new variable-rate image compression framework, which employs generalized octave convolutions (GoConv) and generalized octave transposed-convolutions (GoTConv) To enable a single model to operate with different bit rates and to learn multi-rate image features, a new objective function is introduced. Experimental results show that the proposed framework trained with variable-rate objective function outperforms the standard codecs such as H.265/HEVC-based BPG and state-of-the-art learning-based variable-rate methods.
  arXiv  Detail & Related papers  (2020-12-31T06:26:56Z)
• ALF: Autoencoder-based Low-rank Filter-sharing for Efficient Convolutional Neural Networks [63.91384986073851]
  We propose the autoencoder-based low-rank filter-sharing technique technique (ALF) ALF shows a reduction of 70% in network parameters, 61% in operations and 41% in execution time, with minimal loss in accuracy.
  arXiv  Detail & Related papers  (2020-07-27T09:01:22Z)
• Convolutional neural networks compression with low rank and sparse tensor decompositions [0.0]
  Convolutional neural networks show outstanding results in a variety of computer vision tasks. For some real-world applications, it is crucial to develop models, which can be fast and light enough to run on edge systems and mobile devices. In this work, we consider a neural network compression method based on tensor decompositions.
  arXiv  Detail & Related papers  (2020-06-11T13:53:18Z)
• Kernel Quantization for Efficient Network Compression [59.55192551370948]
  Kernel Quantization (KQ) aims to efficiently convert any pre-trained full-precision convolutional neural network (CNN) model into a low-precision version without significant performance loss. Inspired by the evolution from weight pruning to filter pruning, we propose to quantize in both kernel and weight level. Experiments on the ImageNet classification task prove that KQ needs 1.05 and 1.62 bits on average in VGG and ResNet18, respectively, to represent each parameter in the convolution layer.
  arXiv  Detail & Related papers  (2020-03-11T08:00:04Z)
• Neural Network Compression Framework for fast model inference [59.65531492759006]
  We present a new framework for neural networks compression with fine-tuning, which we called Neural Network Compression Framework (NNCF) It leverages recent advances of various network compression methods and implements some of them, such as sparsity, quantization, and binarization. The framework can be used within the training samples, which are supplied with it, or as a standalone package that can be seamlessly integrated into the existing training code.
  arXiv  Detail & Related papers  (2020-02-20T11:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.