A Survey of Federated Evaluation in Federated Learning
- URL: http://arxiv.org/abs/2305.08070v2
- Date: Fri, 19 May 2023 06:43:38 GMT
- Title: A Survey of Federated Evaluation in Federated Learning
- Authors: Behnaz Soltani, Yipeng Zhou, Venus Haghighi, John C.S. Lui
- Abstract summary: In traditional machine learning, it is trivial to conduct model evaluation since all data samples are managed centrally by a server.
This is because clients do not expose their original data to preserve data privacy.
Federated evaluation plays a vital role in client selection, incentive mechanism design, malicious attack detection, etc.
- Score: 30.56651008584592
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In traditional machine learning, it is trivial to conduct model evaluation
since all data samples are managed centrally by a server. However, model
evaluation becomes a challenging problem in federated learning (FL), which is
called federated evaluation in this work. This is because clients do not expose
their original data to preserve data privacy. Federated evaluation plays a
vital role in client selection, incentive mechanism design, malicious attack
detection, etc. In this paper, we provide the first comprehensive survey of
existing federated evaluation methods. Moreover, we explore various
applications of federated evaluation for enhancing FL performance and finally
present future research directions by envisioning some challenges.
Related papers
- F-Eval: Asssessing Fundamental Abilities with Refined Evaluation Methods [111.46455901113976]
We propose F-Eval, a bilingual evaluation benchmark to evaluate the fundamental abilities, including expression, commonsense and logic.
For reference-free subjective tasks, we devise new evaluation methods, serving as alternatives to scoring by API models.
arXiv Detail & Related papers (2024-01-26T13:55:32Z) - Federated Unlearning: A Survey on Methods, Design Guidelines, and
Evaluation Metrics [2.9093766645364663]
Federated Unlearning (FU) algorithms efficiently remove specific clients' contributions without full model retraining.
This survey provides background concepts, empirical evidence and practical guidelines to design/implement FU schemes.
arXiv Detail & Related papers (2024-01-10T13:26:19Z) - Data Valuation and Detections in Federated Learning [4.899818550820576]
Federated Learning (FL) enables collaborative model training while preserving the privacy of raw data.
A challenge in this framework is the fair and efficient valuation of data, which is crucial for incentivizing clients to contribute high-quality data in the FL task.
This paper introduces a novel privacy-preserving method for evaluating client contributions and selecting relevant datasets without a pre-specified training algorithm in an FL task.
arXiv Detail & Related papers (2023-11-09T12:01:32Z) - A Comprehensive Study on Model Initialization Techniques Ensuring
Efficient Federated Learning [0.0]
Federated learning(FL) has emerged as a promising paradigm for training machine learning models in a distributed and privacy-preserving manner.
The choice of methods used for models plays a crucial role in the performance, convergence speed, communication efficiency, privacy guarantees of federated learning systems.
Our research meticulously compares, categorizes, and delineates the merits and demerits of each technique, examining their applicability across diverse FL scenarios.
arXiv Detail & Related papers (2023-10-31T23:26:58Z) - A Survey for Federated Learning Evaluations: Goals and Measures [26.120949005265345]
Federated learning (FL) is a novel paradigm for privacy-preserving machine learning.
evaluating FL is challenging due to its interdisciplinary nature and diverse goals, such as utility, efficiency, and security.
We introduce FedEval, an open-source platform that provides a standardized and comprehensive evaluation framework for FL algorithms.
arXiv Detail & Related papers (2023-08-23T00:17:51Z) - A Call to Reflect on Evaluation Practices for Failure Detection in Image
Classification [0.491574468325115]
We present a large-scale empirical study for the first time enabling benchmarking confidence scoring functions.
The revelation of a simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation.
arXiv Detail & Related papers (2022-11-28T12:25:27Z) - Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data.
We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks.
We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
arXiv Detail & Related papers (2021-04-27T18:23:54Z) - Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy
Evaluation Approach [84.02388020258141]
We propose a new framework named ENIGMA for estimating human evaluation scores based on off-policy evaluation in reinforcement learning.
ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation.
Our experiments show that ENIGMA significantly outperforms existing methods in terms of correlation with human evaluation scores.
arXiv Detail & Related papers (2021-02-20T03:29:20Z) - Robustness Gym: Unifying the NLP Evaluation Landscape [91.80175115162218]
Deep neural networks are often brittle when deployed in real-world systems.
Recent research has focused on testing the robustness of such models.
We propose a solution in the form of Robustness Gym, a simple and evaluation toolkit.
arXiv Detail & Related papers (2021-01-13T02:37:54Z) - Toward Understanding the Influence of Individual Clients in Federated
Learning [52.07734799278535]
Federated learning allows clients to jointly train a global model without sending their private data to a central server.
We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
arXiv Detail & Related papers (2020-12-20T14:34:36Z) - A Principled Approach to Data Valuation for Federated Learning [73.19984041333599]
Federated learning (FL) is a popular technique to train machine learning (ML) models on decentralized data sources.
The Shapley value (SV) defines a unique payoff scheme that satisfies many desiderata for a data value notion.
This paper proposes a variant of the SV amenable to FL, which we call the federated Shapley value.
arXiv Detail & Related papers (2020-09-14T04:37:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.