A Survey of Distributed Learning in Cloud, Mobile, and Edge Settings

• URL: http://arxiv.org/abs/2405.15079v1
• Date: Thu, 23 May 2024 22:00:38 GMT
• Title: A Survey of Distributed Learning in Cloud, Mobile, and Edge Settings
• Authors: Madison Threadgill, Andreas Gerstlauer,
• Abstract summary: This survey explores the landscape of distributed learning, encompassing cloud and edge settings. We delve into the core concepts of data and model parallelism, examining how models are partitioned across different dimensions and layers to optimize resource utilization and performance. We analyze various partitioning schemes for different layer types, including fully connected, convolutional, and recurrent layers, highlighting the trade-offs between computational efficiency, communication overhead, and memory constraints.
• Score: 1.0589208420411014
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In the era of deep learning (DL), convolutional neural networks (CNNs), and large language models (LLMs), machine learning (ML) models are becoming increasingly complex, demanding significant computational resources for both inference and training stages. To address this challenge, distributed learning has emerged as a crucial approach, employing parallelization across various devices and environments. This survey explores the landscape of distributed learning, encompassing cloud and edge settings. We delve into the core concepts of data and model parallelism, examining how models are partitioned across different dimensions and layers to optimize resource utilization and performance. We analyze various partitioning schemes for different layer types, including fully connected, convolutional, and recurrent layers, highlighting the trade-offs between computational efficiency, communication overhead, and memory constraints. This survey provides valuable insights for future research and development in this rapidly evolving field by comparing and contrasting distributed learning approaches across diverse contexts.

Related papers

• Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference [2.2231908139555734]
  We propose a general performance modeling methodology and workload analysis of distributed LLM training and inference. We validate our performance predictions with published data from literature and relevant industry vendors (e.g., NVIDIA)
  arXiv  Detail & Related papers  (2024-07-19T19:49:05Z)
• Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey [43.57122822150023]
  This article surveys the literature on algorithms and technologies aimed at achieving efficient communication in large-scale distributed deep learning. We first introduce efficient algorithms for model synchronization and communication data compression in the context of large-scale distributed training. Next, we introduce efficient strategies related to resource allocation and task scheduling for use in distributed training and inference.
  arXiv  Detail & Related papers  (2024-04-09T08:35:04Z)
• Deep Metric Learning for Computer Vision: A Brief Overview [4.980117530293724]
  Objective functions that optimize deep neural networks play a vital role in creating an enhanced feature representation of the input data. Deep Metric Learning seeks to develop methods that aim to measure the similarity between data samples. We will provide an overview of recent progress in this area and discuss state-of-the-art Deep Metric Learning approaches.
  arXiv  Detail & Related papers  (2023-12-01T21:53:36Z)
• Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
  We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
  arXiv  Detail & Related papers  (2023-07-12T16:28:21Z)
• Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning [54.67880602409801]
  In this paper, we study the problem of pre-training world models with abundant in-the-wild videos for efficient learning of visual control tasks. We introduce Contextualized World Models (ContextWM) that explicitly separate context and dynamics modeling. Our experiments show that in-the-wild video pre-training equipped with ContextWM can significantly improve the sample efficiency of model-based reinforcement learning.
  arXiv  Detail & Related papers  (2023-05-29T14:29:12Z)
• Adaptive Parameterization of Deep Learning Models for Federated Learning [85.82002651944254]
  Federated Learning offers a way to train deep neural networks in a distributed fashion. It incurs a communication overhead as the model parameters or gradients need to be exchanged regularly during training. In this paper, we propose to utilise parallel Adapters for Federated Learning.
  arXiv  Detail & Related papers  (2023-02-06T17:30:33Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• Parallel Successive Learning for Dynamic Distributed Model Training over Heterogeneous Wireless Networks [50.68446003616802]
  Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices. We develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions. Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning.
  arXiv  Detail & Related papers  (2022-02-07T05:11:01Z)
• Flexible Parallel Learning in Edge Scenarios: Communication, Computational and Energy Cost [20.508003076947848]
  Fog- and IoT-based scenarios often require combining both approaches. We present a framework for flexible parallel learning (FPL), achieving both data and model parallelism. Our experiments, carried out using state-of-the-art deep-network architectures and large-scale datasets, confirm that FPL allows for an excellent trade-off among computational (hence energy) cost, communication overhead, and learning performance.
  arXiv  Detail & Related papers  (2022-01-19T03:47:04Z)
• Edge-assisted Democratized Learning Towards Federated Analytics [67.44078999945722]
  We show the hierarchical learning structure of the proposed edge-assisted democratized learning mechanism, namely Edge-DemLearn. We also validate Edge-DemLearn as a flexible model training mechanism to build a distributed control and aggregation methodology in regions.
  arXiv  Detail & Related papers  (2020-12-01T11:46:03Z)
• Distributed Training of Deep Learning Models: A Taxonomic Perspective [11.924058430461216]
  Distributed deep learning systems (DDLS) train deep neural network models by utilizing the distributed resources of a cluster. We aim to shine some light on the fundamental principles that are at work when training deep neural networks in a cluster of independent machines.
  arXiv  Detail & Related papers  (2020-07-08T08:56:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.