On Effects of Compression with Hyperdimensional Computing in Distributed
Randomized Neural Networks
- URL: http://arxiv.org/abs/2106.09831v1
- Date: Thu, 17 Jun 2021 22:02:40 GMT
- Title: On Effects of Compression with Hyperdimensional Computing in Distributed
Randomized Neural Networks
- Authors: Antonello Rosato, Massimo Panella, Evgeny Osipov, Denis Kleyko
- Abstract summary: We propose a model for distributed classification based on randomized neural networks and hyperdimensional computing.
In this work, we propose a more flexible approach to compression and compare it to conventional compression algorithms, dimensionality reduction, and quantization techniques.
- Score: 6.25118865553438
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: A change of the prevalent supervised learning techniques is foreseeable in
the near future: from the complex, computational expensive algorithms to more
flexible and elementary training ones. The strong revitalization of randomized
algorithms can be framed in this prospect steering. We recently proposed a
model for distributed classification based on randomized neural networks and
hyperdimensional computing, which takes into account cost of information
exchange between agents using compression. The use of compression is important
as it addresses the issues related to the communication bottleneck, however,
the original approach is rigid in the way the compression is used. Therefore,
in this work, we propose a more flexible approach to compression and compare it
to conventional compression algorithms, dimensionality reduction, and
quantization techniques.
Related papers
- Chain of Compression: A Systematic Approach to Combinationally Compress Convolutional Neural Networks [3.309813585671485]
Convolutional neural networks (CNNs) have achieved significant popularity, but their computational and memory intensity poses challenges for resource-constrained computing systems.
Many approaches like quantization, pruning, early exit, and knowledge distillation have demonstrated the effect of reducing redundancy in neural networks.
We propose the Chain of Compression, which works on the combinational sequence to apply these common techniques to compress the neural network.
arXiv Detail & Related papers (2024-03-26T07:26:00Z) - Streaming Lossless Volumetric Compression of Medical Images Using Gated
Recurrent Convolutional Neural Network [0.0]
This paper introduces a hardware-friendly streaming lossless volumetric compression framework.
We propose a gated recurrent convolutional neural network that combines diverse convolutional structures and fusion gate mechanisms.
Our method exhibits robust generalization ability and competitive compression speed.
arXiv Detail & Related papers (2023-11-27T07:19:09Z) - Bandwidth-efficient Inference for Neural Image Compression [26.87198174202502]
We propose an end-to-end differentiable bandwidth efficient neural inference method with the activation compressed by neural data compression method.
Optimized with existing model quantization methods, low-level task of image compression can achieve up to 19x bandwidth reduction with 6.21x energy saving.
arXiv Detail & Related papers (2023-09-06T09:31:37Z) - Towards a Better Theoretical Understanding of Independent Subnetwork Training [56.24689348875711]
We take a closer theoretical look at Independent Subnetwork Training (IST)
IST is a recently proposed and highly effective technique for solving the aforementioned problems.
We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication.
arXiv Detail & Related papers (2023-06-28T18:14:22Z) - Implicit Neural Representations for Image Compression [103.78615661013623]
Implicit Neural Representations (INRs) have gained attention as a novel and effective representation for various data types.
We propose the first comprehensive compression pipeline based on INRs including quantization, quantization-aware retraining and entropy coding.
We find that our approach to source compression with INRs vastly outperforms similar prior work.
arXiv Detail & Related papers (2021-12-08T13:02:53Z) - Joint Global and Local Hierarchical Priors for Learned Image Compression [30.44884350320053]
Recently, learned image compression methods have shown superior performance compared to the traditional hand-crafted image codecs.
We propose a novel entropy model called Information Transformer (Informer) that exploits both local and global information in a content-dependent manner.
Our experiments demonstrate that Informer improves rate-distortion performance over the state-of-the-art methods on the Kodak and Tecnick datasets.
arXiv Detail & Related papers (2021-12-08T06:17:37Z) - A Linearly Convergent Algorithm for Decentralized Optimization: Sending
Less Bits for Free! [72.31332210635524]
Decentralized optimization methods enable on-device training of machine learning models without a central coordinator.
We propose a new randomized first-order method which tackles the communication bottleneck by applying randomized compression operators.
We prove that our method can solve the problems without any increase in the number of communications compared to the baseline.
arXiv Detail & Related papers (2020-11-03T13:35:53Z) - PowerGossip: Practical Low-Rank Communication Compression in
Decentralized Deep Learning [62.440827696638664]
We introduce a simple algorithm that directly compresses the model differences between neighboring workers.
Inspired by the PowerSGD for centralized deep learning, this algorithm uses power steps to maximize the information transferred per bit.
arXiv Detail & Related papers (2020-08-04T09:14:52Z) - Linear Convergent Decentralized Optimization with Compression [50.44269451541387]
Existing decentralized algorithms with compression mainly focus on compressing DGD-type algorithms.
Motivated by primal-dual algorithms, this paper proposes first underlineLinunderlineEAr convergent.
underlineDecentralized with compression, LEAD.
arXiv Detail & Related papers (2020-07-01T04:35:00Z) - Structured Sparsification with Joint Optimization of Group Convolution
and Channel Shuffle [117.95823660228537]
We propose a novel structured sparsification method for efficient network compression.
The proposed method automatically induces structured sparsity on the convolutional weights.
We also address the problem of inter-group communication with a learnable channel shuffle mechanism.
arXiv Detail & Related papers (2020-02-19T12:03:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.