Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Things

• URL: http://arxiv.org/abs/2103.02762v1
• Date: Wed, 3 Mar 2021 23:55:37 GMT
• Title: Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Things
• Authors: Yansong Gao, Minki Kim, Chandra Thapa, Sharif Abuadbba, Zhi Zhang, Seyit A. Camtepe, Hyoungshick Kim, Surya Nepal
• Abstract summary: Federated learning (FL) and split learning (SL) are state-of-the-art distributed machine learning techniques. Recent FL and SL are combined to form splitfed learning (SFL) to leverage each of their benefits. This work considers FL, SL, and SFL, and mount them on Raspberry Pi devices to evaluate their performance.
• Score: 34.544836653715244
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Federated learning (FL) and split learning (SL) are state-of-the-art distributed machine learning techniques to enable machine learning training without accessing raw data on clients or end devices. However, their \emph{comparative training performance} under real-world resource-restricted Internet of Things (IoT) device settings, e.g., Raspberry Pi, remains barely studied, which, to our knowledge, have not yet been evaluated and compared, rendering inconvenient reference for practitioners. This work firstly provides empirical comparisons of FL and SL in real-world IoT settings regarding (i) learning performance with heterogeneous data distributions and (ii) on-device execution overhead. Our analyses in this work demonstrate that the learning performance of SL is better than FL under an imbalanced data distribution but worse than FL under an extreme non-IID data distribution. Recently, FL and SL are combined to form splitfed learning (SFL) to leverage each of their benefits (e.g., parallel training of FL and lightweight on-device computation requirement of SL). This work then considers FL, SL, and SFL, and mount them on Raspberry Pi devices to evaluate their performance, including training time, communication overhead, power consumption, and memory usage. Besides evaluations, we apply two optimizations. Firstly, we generalize SFL by carefully examining the possibility of a hybrid type of model training at the server-side. The generalized SFL merges sequential (dependent) and parallel (independent) processes of model training and is thus beneficial for a system with large-scaled IoT devices, specifically at the server-side operations. Secondly, we propose pragmatic techniques to substantially reduce the communication overhead by up to four times for the SL and (generalized) SFL.

Related papers

• Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach [67.27031215756121]
  Federated learning (FL) is a machine learning paradigm that targets model training without gathering the local data over various data sources. Standard FL, which employs a single server, can only support a limited number of users, leading to degraded learning capability. In this work, we consider a multi-server FL framework, referred to as emphConfederated Learning (CFL) in order to accommodate a larger number of users.
  arXiv  Detail & Related papers  (2024-02-28T03:27:10Z)
• Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Ugly [62.473245910234304]
  This paper takes a hardware-centric approach to explore how Large Language Models can be brought to modern edge computing systems. We provide a micro-level hardware benchmark, compare the model FLOP utilization to a state-of-the-art data center GPU, and study the network utilization in realistic conditions.
  arXiv  Detail & Related papers  (2023-10-04T20:27:20Z)
• Semi-Federated Learning: Convergence Analysis and Optimization of A Hybrid Learning Framework [70.83511997272457]
  We propose a semi-federated learning (SemiFL) paradigm to leverage both the base station (BS) and devices for a hybrid implementation of centralized learning (CL) and FL. We propose a two-stage algorithm to solve this intractable problem, in which we provide the closed-form solutions to the beamformers.
  arXiv  Detail & Related papers  (2023-10-04T03:32:39Z)
• Semi-Supervised Federated Learning for Keyword Spotting [15.044022869136262]
  Keywords Spotting (KWS) is a critical aspect of audio-based applications on mobile devices and virtual assistants. Recent developments in Federated Learning (FL) have significantly expanded the ability to train machine learning models.
  arXiv  Detail & Related papers  (2023-05-09T00:46:12Z)
• Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence [83.58839320635956]
  Federated Learning (FL) can be used in mobile edge networks to train machine learning models in a distributed manner. Recent FL has been interpreted within a Model-Agnostic Meta-Learning (MAML) framework, which brings FL significant advantages in fast adaptation and convergence over heterogeneous datasets. This paper addresses how much benefit MAML brings to FL and how to maximize such benefit over mobile edge networks.
  arXiv  Detail & Related papers  (2023-03-23T02:42:10Z)
• TorchFL: A Performant Library for Bootstrapping Federated Learning Experiments [4.075095403704456]
  We introduce TorchFL, a performant library for bootstrapping federated learning experiments. TorchFL is built on a bottom-up design using PyTorch and Lightning. Being built on a bottom-up design using PyTorch and Lightning, TorchFL provides ready-to-use abstractions for models, datasets, and FL algorithms.
  arXiv  Detail & Related papers  (2022-11-01T20:31:55Z)
• Online Data Selection for Federated Learning with Limited Storage [53.46789303416799]
  Federated Learning (FL) has been proposed to achieve distributed machine learning among networked devices. The impact of on-device storage on the performance of FL is still not explored. In this work, we take the first step to consider the online data selection for FL with limited on-device storage.
  arXiv  Detail & Related papers  (2022-09-01T03:27:33Z)
• ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity [15.908499928588297]
  In Federated Learning (FL), nodes are orders of magnitude more constrained than traditional server-grade hardware. We propose ZeroFL, a framework that relies on highly sparse operations to accelerate on-device training.
  arXiv  Detail & Related papers  (2022-08-04T07:37:07Z)
• Splitfed learning without client-side synchronization: Analyzing client-side split network portion size to overall performance [4.689140226545214]
  Federated Learning (FL), Split Learning (SL), and SplitFed Learning (SFL) are three recent developments in distributed machine learning. This paper studies SFL without client-side model synchronization. It provides only 1%-2% better accuracy than Multi-head Split Learning on the MNIST test set.
  arXiv  Detail & Related papers  (2021-09-19T22:57:23Z)
• Wireless Communications for Collaborative Federated Learning [160.82696473996566]
  Internet of Things (IoT) devices may not be able to transmit their collected data to a central controller for training machine learning models. Google's seminal FL algorithm requires all devices to be directly connected with a central controller. This paper introduces a novel FL framework, called collaborative FL (CFL), which enables edge devices to implement FL with less reliance on a central controller.
  arXiv  Detail & Related papers  (2020-06-03T20:00:02Z)
• SplitFed: When Federated Learning Meets Split Learning [16.212941272007285]
  Federated learning (FL) and split learning (SL) are two popular distributed machine learning approaches. This paper presents a novel approach, named splitfed learning (SFL), that amalgamates the two approaches. SFL provides similar test accuracy and communication efficiency as SL while significantly decreasing its computation time per global epoch than in SL for multiple clients.
  arXiv  Detail & Related papers  (2020-04-25T08:52:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.