PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Experts

• URL: http://arxiv.org/abs/2505.08719v1
• Date: Tue, 13 May 2025 16:27:07 GMT
• Title: PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Experts
• Authors: Yang Su, Na Yan, Yansha Deng, Robert Schober,
• Abstract summary: Large language models (LLMs) hosted on cloud servers alleviate the computational and storage burdens on local devices but raise privacy concerns.<n>Small language models (SLMs) running locally enhance privacy but suffer from limited performance on complex tasks.<n>We propose a privacy-aware wireless collaborative mixture of experts (PWC-MoE) framework to balance computational cost, performance, and privacy protection under bandwidth constraints.
• Score: 59.5243730853157
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) hosted on cloud servers alleviate the computational and storage burdens on local devices but raise privacy concerns due to sensitive data transmission and require substantial communication bandwidth, which is challenging in constrained environments. In contrast, small language models (SLMs) running locally enhance privacy but suffer from limited performance on complex tasks. To balance computational cost, performance, and privacy protection under bandwidth constraints, we propose a privacy-aware wireless collaborative mixture of experts (PWC-MoE) framework. Specifically, PWC-MoE employs a sparse privacy-aware gating network to dynamically route sensitive tokens to privacy experts located on local clients, while non-sensitive tokens are routed to non-privacy experts located at the remote base station. To achieve computational efficiency, the gating network ensures that each token is dynamically routed to and processed by only one expert. To enhance scalability and prevent overloading of specific experts, we introduce a group-wise load-balancing mechanism for the gating network that evenly distributes sensitive tokens among privacy experts and non-sensitive tokens among non-privacy experts. To adapt to bandwidth constraints while preserving model performance, we propose a bandwidth-adaptive and importance-aware token offloading scheme. This scheme incorporates an importance predictor to evaluate the importance scores of non-sensitive tokens, prioritizing the most important tokens for transmission to the base station based on their predicted importance and the available bandwidth. Experiments demonstrate that the PWC-MoE framework effectively preserves privacy and maintains high performance even in bandwidth-constrained environments, offering a practical solution for deploying LLMs in privacy-sensitive and bandwidth-limited scenarios.

Related papers

• SelectiveShield: Lightweight Hybrid Defense Against Gradient Leakage in Federated Learning [4.501710235227319]
  Federated Learning (FL) enables collaborative model training on decentralized data but remains vulnerable to gradient leakage attacks.<n>Existing defense mechanisms, such as differential privacy (DP) and homomorphic encryption (HE), often introduce a trade-off between privacy, model utility, and system overhead.<n>We propose SelectiveShield, a lightweight hybrid defense framework that adaptively integrates homomorphic encryption and differential privacy.
  arXiv  Detail & Related papers  (2025-08-06T09:50:39Z)
• Communication-Efficient and Privacy-Adaptable Mechanism for Federated Learning [33.267664801299354]
  Training machine learning models on decentralized private data via federated learning (FL) poses two key challenges: communication efficiency and privacy protection.<n>We introduce a novel approach called the Communication-Efficient and Privacy-Adaptable Mechanism (CEPAM), achieving both objectives simultaneously.<n>We analyze the trade-offs among user privacy, global utility, and transmission rate of CEPAM by defining appropriate metrics for FL with differential privacy and compression.
  arXiv  Detail & Related papers  (2025-01-21T11:16:05Z)
• Providing Differential Privacy for Federated Learning Over Wireless: A Cross-layer Framework [19.381425127772054]
  Federated Learning (FL) is a distributed machine learning framework that inherently allows edge devices to maintain their local training data.<n>We propose a wireless physical layer (PHY) design for OTA-FL which improves differential privacy (DP) through a decentralized, dynamic power control.<n>This adaptation showcases the flexibility and effectiveness of our design across different learning algorithms while maintaining a strong emphasis on privacy.
  arXiv  Detail & Related papers  (2024-12-05T18:27:09Z)
• Collaborative Inference over Wireless Channels with Feature Differential Privacy [57.68286389879283]
  Collaborative inference among multiple wireless edge devices has the potential to significantly enhance Artificial Intelligence (AI) applications. transmitting extracted features poses a significant privacy risk, as sensitive personal data can be exposed during the process. We propose a novel privacy-preserving collaborative inference mechanism, wherein each edge device in the network secures the privacy of extracted features before transmitting them to a central server for inference.
  arXiv  Detail & Related papers  (2024-10-25T18:11:02Z)
• Enhancing Feature-Specific Data Protection via Bayesian Coordinate Differential Privacy [55.357715095623554]
  Local Differential Privacy (LDP) offers strong privacy guarantees without requiring users to trust external parties. We propose a Bayesian framework, Bayesian Coordinate Differential Privacy (BCDP), that enables feature-specific privacy quantification.
  arXiv  Detail & Related papers  (2024-10-24T03:39:55Z)
• Federated Instruction Tuning of LLMs with Domain Coverage Augmentation [87.49293964617128]
  Federated Domain-specific Instruction Tuning (FedDIT) utilizes limited cross-client private data together with various strategies of instruction augmentation.<n>We propose FedDCA, which optimize domain coverage through greedy client center selection and retrieval-based augmentation.<n>For client-side computational efficiency and system scalability, FedDCA$*$, the variant of FedDCA, utilizes heterogeneous encoders with server-side feature alignment.
  arXiv  Detail & Related papers  (2024-09-30T09:34:31Z)
• Confidential Federated Computations [16.415880530250092]
  Federated Learning and Analytics (FLA) have seen widespread adoption by technology platforms for processing sensitive on-device data.<n>FLA systems do not necessarily require anonymization mechanisms like differential privacy (DP)<n>This paper introduces a novel system architecture that leverages trusted execution environments (TEEs) and open-sourcing to ensure confidentiality of server-side computations.
  arXiv  Detail & Related papers  (2024-04-16T17:47:27Z)
• Libertas: Privacy-Preserving Collective Computation for Decentralised Personal Data Stores [18.91869691495181]
  We introduce a modular architecture, Libertas, to integrate MPC with PDS like Solid.<n>We introduce a paradigm shift from an omniscient' view to individual-based, user-centric view of trust and security.
  arXiv  Detail & Related papers  (2023-09-28T12:07:40Z)
• Distributed Reinforcement Learning for Privacy-Preserving Dynamic Edge Caching [91.50631418179331]
  A privacy-preserving distributed deep policy gradient (P2D3PG) is proposed to maximize the cache hit rates of devices in the MEC networks. We convert the distributed optimizations into model-free Markov decision process problems and then introduce a privacy-preserving federated learning method for popularity prediction.
  arXiv  Detail & Related papers  (2021-10-20T02:48:27Z)
• Federated Intrusion Detection for IoT with Heterogeneous Cohort Privacy [0.0]
  Internet of Things (IoT) devices are becoming increasingly popular and are influencing many application domains such as healthcare and transportation. In this work, we look at differentially private (DP) neural network (NN) based network intrusion detection systems (NIDS) to detect intrusion attacks on networks of such IoT devices. Existing NN training solutions in this domain either ignore privacy considerations or assume that the privacy requirements are homogeneous across all users. We show that the performance of existing differentially private methods degrade for clients with non-identical data distributions when clients' privacy requirements are heterogeneous.
  arXiv  Detail & Related papers  (2021-01-25T03:33:27Z)
• CryptoSPN: Privacy-preserving Sum-Product Network Inference [84.88362774693914]
  We present a framework for privacy-preserving inference of sum-product networks (SPNs) CryptoSPN achieves highly efficient and accurate inference in the order of seconds for medium-sized SPNs.
  arXiv  Detail & Related papers  (2020-02-03T14:49:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.