On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks

• URL: http://arxiv.org/abs/2106.09831v1
• Date: Thu, 17 Jun 2021 22:02:40 GMT
• Title: On Effects of Compression with Hyperdimensional Computing in Distributed Randomized Neural Networks
• Authors: Antonello Rosato, Massimo Panella, Evgeny Osipov, Denis Kleyko
• Abstract summary: We propose a model for distributed classification based on randomized neural networks and hyperdimensional computing. In this work, we propose a more flexible approach to compression and compare it to conventional compression algorithms, dimensionality reduction, and quantization techniques.
• Score: 6.25118865553438
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A change of the prevalent supervised learning techniques is foreseeable in the near future: from the complex, computational expensive algorithms to more flexible and elementary training ones. The strong revitalization of randomized algorithms can be framed in this prospect steering. We recently proposed a model for distributed classification based on randomized neural networks and hyperdimensional computing, which takes into account cost of information exchange between agents using compression. The use of compression is important as it addresses the issues related to the communication bottleneck, however, the original approach is rigid in the way the compression is used. Therefore, in this work, we propose a more flexible approach to compression and compare it to conventional compression algorithms, dimensionality reduction, and quantization techniques.

Related papers

• Causal Context Adjustment Loss for Learned Image Compression [72.7300229848778]
  In recent years, learned image compression (LIC) technologies have surpassed conventional methods notably in terms of rate-distortion (RD) performance. Most present techniques are VAE-based with an autoregressive entropy model, which obviously promotes the RD performance by utilizing the decoded causal context. In this paper, we make the first attempt in investigating the way to explicitly adjust the causal context with our proposed Causal Context Adjustment loss.
  arXiv  Detail & Related papers  (2024-10-07T09:08:32Z)
• AlphaZip: Neural Network-Enhanced Lossless Text Compression [0.0]
  This paper introduces a lossless text compression approach using a Large Language Model (LLM) The method involves two key steps: first, prediction using a dense neural network architecture, such as a transformer block; second, compressing the predicted ranks with standard compression algorithms like Adaptive Huffman, LZ77, or Gzip.
  arXiv  Detail & Related papers  (2024-09-23T14:21:06Z)
• Streaming Lossless Volumetric Compression of Medical Images Using Gated Recurrent Convolutional Neural Network [0.0]
  This paper introduces a hardware-friendly streaming lossless volumetric compression framework. We propose a gated recurrent convolutional neural network that combines diverse convolutional structures and fusion gate mechanisms. Our method exhibits robust generalization ability and competitive compression speed.
  arXiv  Detail & Related papers  (2023-11-27T07:19:09Z)
• Bandwidth-efficient Inference for Neural Image Compression [26.87198174202502]
  We propose an end-to-end differentiable bandwidth efficient neural inference method with the activation compressed by neural data compression method. Optimized with existing model quantization methods, low-level task of image compression can achieve up to 19x bandwidth reduction with 6.21x energy saving.
  arXiv  Detail & Related papers  (2023-09-06T09:31:37Z)
• Towards a Better Theoretical Understanding of Independent Subnetwork Training [56.24689348875711]
  We take a closer theoretical look at Independent Subnetwork Training (IST) IST is a recently proposed and highly effective technique for solving the aforementioned problems. We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication.
  arXiv  Detail & Related papers  (2023-06-28T18:14:22Z)
• Implicit Neural Representations for Image Compression [103.78615661013623]
  Implicit Neural Representations (INRs) have gained attention as a novel and effective representation for various data types. We propose the first comprehensive compression pipeline based on INRs including quantization, quantization-aware retraining and entropy coding. We find that our approach to source compression with INRs vastly outperforms similar prior work.
  arXiv  Detail & Related papers  (2021-12-08T13:02:53Z)
• Joint Global and Local Hierarchical Priors for Learned Image Compression [30.44884350320053]
  Recently, learned image compression methods have shown superior performance compared to the traditional hand-crafted image codecs. We propose a novel entropy model called Information Transformer (Informer) that exploits both local and global information in a content-dependent manner. Our experiments demonstrate that Informer improves rate-distortion performance over the state-of-the-art methods on the Kodak and Tecnick datasets.
  arXiv  Detail & Related papers  (2021-12-08T06:17:37Z)
• A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free! [72.31332210635524]
  Decentralized optimization methods enable on-device training of machine learning models without a central coordinator. We propose a new randomized first-order method which tackles the communication bottleneck by applying randomized compression operators. We prove that our method can solve the problems without any increase in the number of communications compared to the baseline.
  arXiv  Detail & Related papers  (2020-11-03T13:35:53Z)
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning [62.440827696638664]
  We introduce a simple algorithm that directly compresses the model differences between neighboring workers. Inspired by the PowerSGD for centralized deep learning, this algorithm uses power steps to maximize the information transferred per bit.
  arXiv  Detail & Related papers  (2020-08-04T09:14:52Z)
• Structured Sparsification with Joint Optimization of Group Convolution and Channel Shuffle [117.95823660228537]
  We propose a novel structured sparsification method for efficient network compression. The proposed method automatically induces structured sparsity on the convolutional weights. We also address the problem of inter-group communication with a learnable channel shuffle mechanism.
  arXiv  Detail & Related papers  (2020-02-19T12:03:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.