Lottery Hypothesis based Unsupervised Pre-training for Model Compression in Federated Learning

• URL: http://arxiv.org/abs/2004.09817v1
• Date: Tue, 21 Apr 2020 08:31:23 GMT
• Title: Lottery Hypothesis based Unsupervised Pre-training for Model Compression in Federated Learning
• Authors: Sohei Itahara, Takayuki Nishio, Masahiro Morikura and Koji Yamamoto
• Abstract summary: Federated learning (FL) enables a neural network (NN) to be trained using privacy-sensitive data on mobile devices. This paper proposes a novel unsupervised pre-training method adapted for FL. It aims to reduce both the communication and computation costs through model compression.
• Score: 8.90077503980675
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) enables a neural network (NN) to be trained using privacy-sensitive data on mobile devices while retaining all the data on their local storages. However, FL asks the mobile devices to perform heavy communication and computation tasks, i.e., devices are requested to upload and download large-volume NN models and train them. This paper proposes a novel unsupervised pre-training method adapted for FL, which aims to reduce both the communication and computation costs through model compression. Since the communication and computation costs are highly dependent on the volume of NN models, reducing the volume without decreasing model performance can reduce these costs. The proposed pre-training method leverages unlabeled data, which is expected to be obtained from the Internet or data repository much more easily than labeled data. The key idea of the proposed method is to obtain a ``good'' subnetwork from the original NN using the unlabeled data based on the lottery hypothesis. The proposed method trains an original model using a denoising auto encoder with the unlabeled data and then prunes small-magnitude parameters of the original model to generate a small but good subnetwork. The proposed method is evaluated using an image classification task. The results show that the proposed method requires 35\% less traffic and computation time than previous methods when achieving a certain test accuracy.

Related papers

• Just How Flexible are Neural Networks in Practice? [89.80474583606242]
  It is widely believed that a neural network can fit a training set containing at least as many samples as it has parameters. In practice, however, we only find solutions via our training procedure, including the gradient and regularizers, limiting flexibility.
  arXiv  Detail & Related papers  (2024-06-17T12:24:45Z)
• Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Models [17.34908967455907]
  machine unlearning'' proposes the selective removal of unwanted data without the need for retraining from scratch. Fast-NTK is a novel NTK-based unlearning algorithm that significantly reduces the computational complexity.
  arXiv  Detail & Related papers  (2023-12-22T18:55:45Z)
• Fast, Distribution-free Predictive Inference for Neural Networks with Coverage Guarantees [25.798057062452443]
  This paper introduces a novel, computationally-efficient algorithm for predictive inference (PI) It requires no distributional assumptions on the data and can be computed faster than existing bootstrap-type methods for neural networks.
  arXiv  Detail & Related papers  (2023-06-11T04:03:58Z)
• Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogramming [129.4950757742912]
  We introduce a novel method for leveraging pre-trained models for low-resource (music) classification based on the concept of Neural Model Reprogramming (NMR) NMR aims at re-purposing a pre-trained model from a source domain to a target domain by modifying the input of a frozen pre-trained model. Experimental results suggest that a neural model pre-trained on large-scale datasets can successfully perform music genre classification by using this reprogramming method.
  arXiv  Detail & Related papers  (2022-11-02T17:38:33Z)
• Learning from Data with Noisy Labels Using Temporal Self-Ensemble [11.245833546360386]
  Deep neural networks (DNNs) have an enormous capacity to memorize noisy labels. Current state-of-the-art methods present a co-training scheme that trains dual networks using samples associated with small losses. We propose a simple yet effective robust training scheme that operates by training only a single network.
  arXiv  Detail & Related papers  (2022-07-21T08:16:31Z)
• Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices [14.604785223644718]
  Federated Learning (FL) is a technique to train models using data distributed across devices. Differential Privacy (DP) provides a formal privacy guarantee for sensitive data. We propose Partial Embedding Updates (PEU) to decrease noise by decreasing payload size.
  arXiv  Detail & Related papers  (2022-07-18T23:53:17Z)
• One-Pixel Shortcut: on the Learning Preference of Deep Neural Networks [28.502489028888608]
  Unlearnable examples (ULEs) aim to protect data from unauthorized usage for training DNNs. In adversarial training, the unlearnability of error-minimizing noise will severely degrade. We propose a novel model-free method, named emphOne-Pixel Shortcut, which only perturbs a single pixel of each image and makes the dataset unlearnable.
  arXiv  Detail & Related papers  (2022-05-24T15:17:52Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)
• Probing Model Signal-Awareness via Prediction-Preserving Input Minimization [67.62847721118142]
  We evaluate models' ability to capture the correct vulnerability signals to produce their predictions. We measure the signal awareness of models using a new metric we propose- Signal-aware Recall (SAR) The results show a sharp drop in the model's Recall from the high 90s to sub-60s with the new metric.
  arXiv  Detail & Related papers  (2020-11-25T20:05:23Z)
• UVeQFed: Universal Vector Quantization for Federated Learning [179.06583469293386]
  Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their possibly private labeled data. In FL, each user trains its copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion.
  arXiv  Detail & Related papers  (2020-06-05T07:10:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.