Split Knowledge Distillation for Large Models in IoT: Architecture, Challenges, and Solutions

• URL: http://arxiv.org/abs/2501.17164v1
• Date: Tue, 17 Dec 2024 02:31:31 GMT
• Title: Split Knowledge Distillation for Large Models in IoT: Architecture, Challenges, and Solutions
• Authors: Zuguang Li, Wen Wu, Shaohua Wu, Qiaohua Lin, Yaping Sun, Hui Wang,
• Abstract summary: Large models (LMs) have immense potential in Internet of Things (IoT) systems, enabling applications such as intelligent voice assistants, predictive maintenance, and healthcare monitoring.<n>Training LMs on edge servers raises data privacy concerns, while deploying them directly on IoT devices is constrained by limited computational and memory resources.<n>We propose a split knowledge distillation framework to efficiently distill LMs into smaller, deployable versions for IoT devices while ensuring raw data remains local.
• Score: 16.25411682771788
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large models (LMs) have immense potential in Internet of Things (IoT) systems, enabling applications such as intelligent voice assistants, predictive maintenance, and healthcare monitoring. However, training LMs on edge servers raises data privacy concerns, while deploying them directly on IoT devices is constrained by limited computational and memory resources. We analyze the key challenges of training LMs in IoT systems, including energy constraints, latency requirements, and device heterogeneity, and propose potential solutions such as dynamic resource management, adaptive model partitioning, and clustered collaborative training. Furthermore, we propose a split knowledge distillation framework to efficiently distill LMs into smaller, deployable versions for IoT devices while ensuring raw data remains local. This framework integrates knowledge distillation and split learning to minimize energy consumption and meet low model training delay requirements. A case study is presented to evaluate the feasibility and performance of the proposed framework.

Related papers

• LLMs meet Federated Learning for Scalable and Secure IoT Management [6.649910168731417]
  Traditional centralized architectures struggle with latency, privacy concerns, and excessive resource consumption. This paper presents a novel Federated Learning-driven Large Language Model (FL-LLM) framework, designed to enhance IoT system intelligence.
  arXiv  Detail & Related papers  (2025-04-22T16:56:59Z)
• Deploying Large AI Models on Resource-Limited Devices with Split Federated Learning [39.73152182572741]
  This paper proposes a novel framework, named Quantized Split Federated Fine-Tuning Large AI Model (SFLAM) By partitioning the training load between edge devices and servers, SFLAM can facilitate the operation of large models on devices. SFLAM incorporates quantization management, power control, and bandwidth allocation strategies to enhance training efficiency.
  arXiv  Detail & Related papers  (2025-04-12T07:55:11Z)
• Speech Emotion Recognition under Resource Constraints with Data Distillation [64.36799373890916]
  Speech emotion recognition (SER) plays a crucial role in human-computer interaction. The emergence of edge devices in the Internet of Things presents challenges in constructing intricate deep learning models. We propose a data distillation framework to facilitate efficient development of SER models in IoT applications.
  arXiv  Detail & Related papers  (2024-06-21T13:10:46Z)
• Effective Intrusion Detection in Heterogeneous Internet-of-Things Networks via Ensemble Knowledge Distillation-based Federated Learning [52.6706505729803]
  We introduce Federated Learning (FL) to collaboratively train a decentralized shared model of Intrusion Detection Systems (IDS) FLEKD enables a more flexible aggregation method than conventional model fusion techniques. Experiment results show that the proposed approach outperforms local training and traditional FL in terms of both speed and performance.
  arXiv  Detail & Related papers  (2024-01-22T14:16:37Z)
• Edge-assisted U-Shaped Split Federated Learning with Privacy-preserving for Internet of Things [4.68267059122563]
  We present an innovative Edge-assisted U-Shaped Split Federated Learning (EUSFL) framework, which harnesses the high-performance capabilities of edge servers. In this framework, we leverage Federated Learning (FL) to enable data holders to collaboratively train models without sharing their data. We also propose a novel noise mechanism called LabelDP to ensure that data features and labels can securely resist reconstruction attacks.
  arXiv  Detail & Related papers  (2023-11-08T05:14:41Z)
• Filling the Missing: Exploring Generative AI for Enhanced Federated Learning over Heterogeneous Mobile Edge Devices [72.61177465035031]
  We propose a generative AI-empowered federated learning to address these challenges by leveraging the idea of FIlling the MIssing (FIMI) portion of local data. Experiment results demonstrate that FIMI can save up to 50% of the device-side energy to achieve the target global test accuracy.
  arXiv  Detail & Related papers  (2023-10-21T12:07:04Z)
• Federated Learning in IoT: a Survey from a Resource-Constrained Perspective [0.0]
  Federated Learning (FL), a decentralized machine learning technique, is widely used to collect and train machine learning models from a variety of distributed data sources. However, the resource-constrained nature of IoT devices prevents the widescale deployment FL in the real world. This research paper presents a comprehensive survey of the challenges and solutions associated with implementing Federated Learning (FL) in resource-constrained Internet of Things (IoT) environments.
  arXiv  Detail & Related papers  (2023-08-25T03:31:22Z)
• Computational Intelligence and Deep Learning for Next-Generation Edge-Enabled Industrial IoT [51.68933585002123]
  We investigate how to deploy computational intelligence and deep learning (DL) in edge-enabled industrial IoT networks. In this paper, we propose a novel multi-exit-based federated edge learning (ME-FEEL) framework. In particular, the proposed ME-FEEL can achieve an accuracy gain up to 32.7% in the industrial IoT networks with the severely limited resources.
  arXiv  Detail & Related papers  (2021-10-28T08:14:57Z)
• Energy-Efficient Multi-Orchestrator Mobile Edge Learning [54.28419430315478]
  Mobile Edge Learning (MEL) is a collaborative learning paradigm that features distributed training of Machine Learning (ML) models over edge devices. In MEL, possible coexistence of multiple learning tasks with different datasets may arise. We propose lightweight algorithms that can achieve near-optimal performance and facilitate the trade-offs between energy consumption, accuracy, and solution complexity.
  arXiv  Detail & Related papers  (2021-09-02T07:37:10Z)
• Reliable Fleet Analytics for Edge IoT Solutions [0.0]
  We propose a framework for facilitating machine learning at the edge for AIoT applications. The contribution is an architecture that includes services, tools, and methods for delivering fleet analytics at scale. We present a preliminary validation of the framework by performing experiments with IoT devices on a university campus's rooms.
  arXiv  Detail & Related papers  (2021-01-12T11:28:43Z)
• Optimizing Resource-Efficiency for Federated Edge Intelligence in IoT Networks [96.24723959137218]
  We study an edge intelligence-based IoT network in which a set of edge servers learn a shared model using federated learning (FL) We propose a novel framework, called federated edge intelligence (FEI), that allows edge servers to evaluate the required number of data samples according to the energy cost of the IoT network. We prove that our proposed algorithm does not cause any data leakage nor disclose any topological information of the IoT network.
  arXiv  Detail & Related papers  (2020-11-25T12:51:59Z)
• LIMITS: Lightweight Machine Learning for IoT Systems with Resource Limitations [8.647853543335662]
  We present the novel open source framework LIghtweight Machine learning for IoT Systems (LIMITS) LIMITS applies a platform-in-the-loop approach explicitly considering the actual compilation toolchain of the target IoT platform. We apply and validate LIMITS in two case studies focusing on cellular data rate prediction and radio-based vehicle classification.
  arXiv  Detail & Related papers  (2020-01-28T06:34:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.