FedEval: A Benchmark System with a Comprehensive Evaluation Model for
Federated Learning
- URL: http://arxiv.org/abs/2011.09655v2
- Date: Wed, 25 Nov 2020 16:08:13 GMT
- Title: FedEval: A Benchmark System with a Comprehensive Evaluation Model for
Federated Learning
- Authors: Di Chai and Leye Wang and Kai Chen and Qiang Yang
- Abstract summary: In this paper, we propose a comprehensive evaluation framework for federated learning (FL) systems.
We first introduce the ACTPR model, which defines five metrics that cannot be excluded in FL evaluation, including Accuracy, Communication, Time efficiency, Privacy, and Robustness.
We then provide an in-depth benchmarking study between the two most widely-used FL mechanisms, FedSGD and FedAvg.
- Score: 17.680627081257246
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: As an innovative solution for privacy-preserving machine learning (ML),
federated learning (FL) is attracting much attention from research and industry
areas. While new technologies proposed in the past few years do evolve the FL
area, unfortunately, the evaluation results presented in these works fall short
in integrity and are hardly comparable because of the inconsistent evaluation
metrics and the lack of a common platform. In this paper, we propose a
comprehensive evaluation framework for FL systems. Specifically, we first
introduce the ACTPR model, which defines five metrics that cannot be excluded
in FL evaluation, including Accuracy, Communication, Time efficiency, Privacy,
and Robustness. Then we design and implement a benchmarking system called
FedEval, which enables the systematic evaluation and comparison of existing
works under consistent experimental conditions. We then provide an in-depth
benchmarking study between the two most widely-used FL mechanisms, FedSGD and
FedAvg. The benchmarking results show that FedSGD and FedAvg both have
advantages and disadvantages under the ACTPR model. For example, FedSGD is
barely influenced by the none independent and identically distributed (non-IID)
data problem, but FedAvg suffers from a decline in accuracy of up to 9% in our
experiments. On the other hand, FedAvg is more efficient than FedSGD regarding
time consumption and communication. Lastly, we excavate a set of take-away
conclusions, which are very helpful for researchers in the FL area.
Related papers
- Which mode is better for federated learning? Centralized or
Decentralized [64.46017397813549]
Both centralized and decentralized approaches have shown excellent performance and great application value in federated learning (FL)
However, current studies do not provide evidence to show which one performs better.
arXiv Detail & Related papers (2023-10-05T11:09:42Z) - FedVal: Different good or different bad in federated learning [9.558549875692808]
Federated learning (FL) systems are susceptible to attacks from malicious actors.
FL poses new challenges in addressing group bias, such as ensuring fair performance for different demographic groups.
Traditional methods used to address such biases require centralized access to the data, which FL systems do not have.
We present a novel approach FedVal for both robustness and fairness that does not require any additional information from clients.
arXiv Detail & Related papers (2023-06-06T22:11:13Z) - Reliable Federated Disentangling Network for Non-IID Domain Feature [62.73267904147804]
In this paper, we propose a novel reliable federated disentangling network, termed RFedDis.
To the best of our knowledge, our proposed RFedDis is the first work to develop an FL approach based on evidential uncertainty combined with feature disentangling.
Our proposed RFedDis provides outstanding performance with a high degree of reliability as compared to other state-of-the-art FL approaches.
arXiv Detail & Related papers (2023-01-30T11:46:34Z) - FedSkip: Combatting Statistical Heterogeneity with Federated Skip
Aggregation [95.85026305874824]
We introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices.
We conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiency.
arXiv Detail & Related papers (2022-12-14T13:57:01Z) - Communication-Efficient Diffusion Strategy for Performance Improvement
of Federated Learning with Non-IID Data [10.112913394578703]
Federated learning (FL) is a novel learning paradigm that addresses the privacy leakage challenge of centralized learning.
In FL, users with non-independent and identically distributed (non-IID) characteristics can deteriorate the performance of the global model.
We propose a novel diffusion strategy of the machine learning (ML) model (FedDif) to maximize the FL performance with non-IID data.
arXiv Detail & Related papers (2022-07-15T14:28:41Z) - A Closer Look at Debiased Temporal Sentence Grounding in Videos:
Dataset, Metric, and Approach [53.727460222955266]
Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video.
Recent studies have found that current benchmark datasets may have obvious moment annotation biases.
We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
arXiv Detail & Related papers (2022-03-10T08:58:18Z) - FedPrune: Towards Inclusive Federated Learning [1.308951527147782]
Federated learning (FL) is a distributed learning technique that trains a shared model over distributed data in a privacy-preserving manner.
We propose FedPrune; a system that tackles this challenge by pruning the global model for slow clients based on their device characteristics.
By using insights from Central Limit Theorem, FedPrune incorporates a new aggregation technique that achieves robust performance over non-IID data.
arXiv Detail & Related papers (2021-10-27T06:33:38Z) - FedScale: Benchmarking Model and System Performance of Federated
Learning [4.1617240682257925]
FedScale is a set of challenging and realistic benchmark datasets for federated learning (FL) research.
FedScale is open-source with permissive licenses and actively maintained.
arXiv Detail & Related papers (2021-05-24T15:55:27Z) - FedSemi: An Adaptive Federated Semi-Supervised Learning Framework [23.90642104477983]
Federated learning (FL) has emerged as an effective technique to co-training machine learning models without actually sharing data and leaking privacy.
Most existing FL methods focus on the supervised setting and ignore the utilization of unlabeled data.
We propose FedSemi, a novel, adaptive, and general framework, which firstly introduces the consistency regularization into FL using a teacher-student model.
arXiv Detail & Related papers (2020-12-06T15:46:04Z) - Stochastic-Sign SGD for Federated Learning with Theoretical Guarantees [49.91477656517431]
Quantization-based solvers have been widely adopted in Federated Learning (FL)
No existing methods enjoy all the aforementioned properties.
We propose an intuitively-simple yet theoretically-simple method based on SIGNSGD to bridge the gap.
arXiv Detail & Related papers (2020-02-25T15:12:15Z) - FedDANE: A Federated Newton-Type Method [49.9423212899788]
Federated learning aims to jointly learn low statistical models over massively distributed datasets.
We propose FedDANE, an optimization that we adapt from DANE, to handle federated learning.
arXiv Detail & Related papers (2020-01-07T07:44:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.