Deep Hierarchy Quantization Compression algorithm based on Dynamic Sampling

• URL: http://arxiv.org/abs/2212.14760v1
• Date: Fri, 30 Dec 2022 15:12:30 GMT
• Title: Deep Hierarchy Quantization Compression algorithm based on Dynamic Sampling
• Authors: Wan Jiang, Gang Liu, Xiaofeng Chen, Yipeng Zhou
• Abstract summary: Federated machine learning stores data locally for training and aggregates the models on the server. During the training process, the transmission of model parameters can impose a significant load on the network bandwidth. We propose a deep hierarchical quantization compression algorithm, which further compresses the model and reduces the network load brought by data transmission.
• Score: 11.439540966972212
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unlike traditional distributed machine learning, federated learning stores data locally for training and then aggregates the models on the server, which solves the data security problem that may arise in traditional distributed machine learning. However, during the training process, the transmission of model parameters can impose a significant load on the network bandwidth. It has been pointed out that the vast majority of model parameters are redundant during model parameter transmission. In this paper, we explore the data distribution law of selected partial model parameters on this basis, and propose a deep hierarchical quantization compression algorithm, which further compresses the model and reduces the network load brought by data transmission through the hierarchical quantization of model parameters. And we adopt a dynamic sampling strategy for the selection of clients to accelerate the convergence of the model. Experimental results on different public datasets demonstrate the effectiveness of our algorithm.

Related papers

• Transferable Post-training via Inverse Value Learning [83.75002867411263]
  We propose modeling changes at the logits level during post-training using a separate neural network (i.e., the value network) After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes.
  arXiv  Detail & Related papers  (2024-10-28T13:48:43Z)
• Variational autoencoder-based neural network model compression [4.992476489874941]
  Variational Autoencoders (VAEs), as a form of deep generative model, have been widely used in recent years. This paper aims to explore neural network model compression method based on VAE.
  arXiv  Detail & Related papers  (2024-08-25T09:06:22Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Learning to Learn with Generative Models of Neural Network Checkpoints [71.06722933442956]
  We construct a dataset of neural network checkpoints and train a generative model on the parameters. We find that our approach successfully generates parameters for a wide range of loss prompts. We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.
  arXiv  Detail & Related papers  (2022-09-26T17:59:58Z)
• On the Influence of Enforcing Model Identifiability on Learning dynamics of Gaussian Mixture Models [14.759688428864159]
  We propose a technique for extracting submodels from singular models. Our method enforces model identifiability during training. We show how the method can be applied to more complex models like deep neural networks.
  arXiv  Detail & Related papers  (2022-06-17T07:50:22Z)
• Online Model Compression for Federated Learning with Large Models [8.48327410170884]
  Online Model Compression (OMC) is a framework that stores model parameters in a compressed format and decompresses them only when needed. OMC can reduce memory usage and communication cost of model parameters by up to 59% while attaining comparable accuracy and training speed when compared with full-precision training.
  arXiv  Detail & Related papers  (2022-05-06T22:43:03Z)
• Dynamically-Scaled Deep Canonical Correlation Analysis [77.34726150561087]
  Canonical Correlation Analysis (CCA) is a method for feature extraction of two views by finding maximally correlated linear projections of them. We introduce a novel dynamic scaling method for training an input-dependent canonical correlation model.
  arXiv  Detail & Related papers  (2022-03-23T12:52:49Z)
• Prediction-Centric Learning of Independent Cascade Dynamics from Partial Observations [13.680949377743392]
  We address the problem of learning of a spreading model such that the predictions generated from this model are accurate. We introduce a computationally efficient algorithm, based on a scalable dynamic message-passing approach. We show that tractable inference from the learned model generates a better prediction of marginal probabilities compared to the original model.
  arXiv  Detail & Related papers  (2020-07-13T17:58:21Z)
• Learning Queuing Networks by Recurrent Neural Networks [0.0]
  We propose a machine-learning approach to derive performance models from data. We exploit a deterministic approximation of their average dynamics in terms of a compact system of ordinary differential equations. This allows for an interpretable structure of the neural network, which can be trained from system measurements to yield a white-box parameterized model.
  arXiv  Detail & Related papers  (2020-02-25T10:56:47Z)
• Model Fusion via Optimal Transport [64.13185244219353]
  We present a layer-wise model fusion algorithm for neural networks. We show that this can successfully yield "one-shot" knowledge transfer between neural networks trained on heterogeneous non-i.i.d. data.
  arXiv  Detail & Related papers  (2019-10-12T22:07:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.