Communication-Efficient Federated Knowledge Graph Embedding with Entity-Wise Top-K Sparsification

• URL: http://arxiv.org/abs/2406.13225v1
• Date: Wed, 19 Jun 2024 05:26:02 GMT
• Title: Communication-Efficient Federated Knowledge Graph Embedding with Entity-Wise Top-K Sparsification
• Authors: Xiaoxiong Zhang, Zhiwei Zeng, Xin Zhou, Dusit Niyato, Zhiqi Shen,
• Abstract summary: Federated Knowledge Graphs Embedding learning (FKGE) encounters challenges in communication efficiency stemming from the considerable size of parameters and extensive communication rounds. We propose bidirectional communication-efficient FedS based on Entity-Wise Top-K Sparsification strategy.
• Score: 49.66272783945571
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Knowledge Graphs Embedding learning (FKGE) encounters challenges in communication efficiency stemming from the considerable size of parameters and extensive communication rounds. However, existing FKGE methods only focus on reducing communication rounds by conducting multiple rounds of local training in each communication round, and ignore reducing the size of parameters transmitted within each communication round. To tackle the problem, we first find that universal reduction in embedding precision across all entities during compression can significantly impede convergence speed, underscoring the importance of maintaining embedding precision. We then propose bidirectional communication-efficient FedS based on Entity-Wise Top-K Sparsification strategy. During upload, clients dynamically identify and upload only the Top-K entity embeddings with the greater changes to the server. During download, the server first performs personalized embedding aggregation for each client. It then identifies and transmits the Top-K aggregated embeddings to each client. Besides, an Intermittent Synchronization Mechanism is used by FedS to mitigate negative effect of embedding inconsistency among shared entities of clients caused by heterogeneity of Federated Knowledge Graph. Extensive experiments across three datasets showcase that FedS significantly enhances communication efficiency with negligible (even no) performance degradation.

Related papers

• Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration [66.43954501171292]
  We introduce Catalyst Acceleration and propose an acceleration Decentralized Federated Learning algorithm called DFedCata. DFedCata consists of two main components: the Moreau envelope function, which addresses parameter inconsistencies, and Nesterov's extrapolation step, which accelerates the aggregation phase. Empirically, we demonstrate the advantages of the proposed algorithm in both convergence speed and generalization performance on CIFAR10/100 with various non-iid data distributions.
  arXiv  Detail & Related papers  (2024-10-09T06:17:16Z)
• Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning [29.727339562140653]
  Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL) These methods encounter challenges such as the straggler problem and diminished model performance due to heterogeneous bandwidth and non-IID data. We introduce a bandwidth-aware compression framework for FL, aimed at improving communication efficiency while mitigating the problems associated with non-IID data.
  arXiv  Detail & Related papers  (2024-08-27T02:28:27Z)
• FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models [56.21666819468249]
  Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with a central server. We introduce FedComLoc, integrating practical and effective compression into emphScaffnew to further enhance communication efficiency.
  arXiv  Detail & Related papers  (2024-03-14T22:29:59Z)
• Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks [66.78640306687227]
  To protect privacy and meet legal regulations, federated learning (FL) has gained significant attention for training speech-to-text (S2T) systems. The commonly used FL approach (i.e., textscFedAvg) in S2T tasks typically suffers from extensive communication overhead. We propose a personalized federated S2T framework that introduces textscFedLoRA, a lightweight LoRA module for client-side tuning and interaction with the server, and textscFedMem, a global model equipped with a $k$-near
  arXiv  Detail & Related papers  (2024-01-18T15:39:38Z)
• FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learning [91.74206675452888]
  We propose a novel method FedFM, which guides each client's features to match shared category-wise anchors. To achieve higher efficiency and flexibility, we propose a FedFM variant, called FedFM-Lite, where clients communicate with server with fewer synchronization times and communication bandwidth costs.
  arXiv  Detail & Related papers  (2022-10-14T08:11:34Z)
• FedGCN: Convergence-Communication Tradeoffs in Federated Training of Graph Convolutional Networks [14.824579000821272]
  We introduce the Federated Graph Convolutional Network (FedGCN) algorithm, which uses federated learning to train GCN models for semi-supervised node classification. Compared to prior methods that require extra communication among clients at each training round, FedGCN clients only communicate with the central server in one pre-training step. Experimental results show that our FedGCN algorithm achieves better model accuracy with 51.7% faster convergence on average and at least 100X less communication compared to prior work.
  arXiv  Detail & Related papers  (2022-01-28T21:39:16Z)
• Decentralized Federated Averaging [17.63112147669365]
  Federated averaging (FedAvg) is a communication efficient algorithm for the distributed training with an enormous number of clients. We study the decentralized FedAvg with momentum (DFedAvgM), which is implemented on clients that are connected by an undirected graph.
  arXiv  Detail & Related papers  (2021-04-23T02:01:30Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)
• Distributed Sparse SGD with Majority Voting [5.32836690371986]
  We introduce a majority voting based sparse communication strategy for distributed learning. We show that it is possible to achieve up to x4000 compression without any loss in the test accuracy.
  arXiv  Detail & Related papers  (2020-11-12T17:06:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.