FLBench: A Benchmark Suite for Federated Learning
- URL: http://arxiv.org/abs/2008.07257v3
- Date: Fri, 12 Mar 2021 02:22:31 GMT
- Title: FLBench: A Benchmark Suite for Federated Learning
- Authors: Yuan Liang, Yange Guo, Yanxia Gong, Chunjie Luo, Jianfeng Zhan, Yunyou
Huang
- Abstract summary: Federated learning is a new machine learning paradigm.
The goal is to build a machine learning model from the data sets distributed on multiple devices so-called an isolated data island.
This paper presents a federated learning benchmark suite named FLBench.
- Score: 7.9325785873650405
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated learning is a new machine learning paradigm. The goal is to build a
machine learning model from the data sets distributed on multiple devices
so-called an isolated data island, while keeping their data secure and private.
Most existing federated learning benchmarks work manually splits commonly used
public datasets into partitions to simulate real world isolated data island
scenarios. Still, this simulation fails to capture real world isolated data
island intrinsic characteristics. This paper presents a federated learning (FL)
benchmark suite named FLBench. FLBench contains three domains: medical,
financial, and AIoT. By configuring various domains, FLBench is qualified to
evaluate federated learning systems and algorithms essential aspects, like
communication, scenario transformation, privacy-preserving, data distribution
heterogeneity, and cooperation strategy. Hence, it becomes a promising platform
for developing novel federated learning algorithms. Currently, FLBench is open
sourced and in fast evolution. We package it as an automated deployment tool.
The benchmark suite is available from
https://www.benchcouncil.org/flbench.html.
Related papers
- FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models [48.484485609995986]
Federated learning has enabled multiple parties to collaboratively train large language models without directly sharing their data (FedLLM)
There are currently no realistic datasets and benchmarks for FedLLM.
We propose FedLLM-Bench, which involves 8 training methods, 4 training datasets, and 6 evaluation metrics.
arXiv Detail & Related papers (2024-06-07T11:19:30Z) - pfl-research: simulation framework for accelerating research in Private Federated Learning [6.421821657238535]
pfl-research is a fast, modular, and easy-to-use Python framework for simulating Federated learning (FL)
It supports setups, PyTorch, and non-neural network models, and is tightly integrated with state-of-the-art algorithms.
We release a suite of benchmarks that evaluates an algorithm's overall performance on a diverse set of realistic scenarios.
arXiv Detail & Related papers (2024-04-09T16:23:01Z) - Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device)
In FL, each data holder trains a model locally and releases it to a central server for aggregation.
In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation).
In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
arXiv Detail & Related papers (2022-11-20T10:49:22Z) - TorchFL: A Performant Library for Bootstrapping Federated Learning
Experiments [4.075095403704456]
We introduce TorchFL, a performant library for bootstrapping federated learning experiments.
TorchFL is built on a bottom-up design using PyTorch and Lightning.
Being built on a bottom-up design using PyTorch and Lightning, TorchFL provides ready-to-use abstractions for models, datasets, and FL algorithms.
arXiv Detail & Related papers (2022-11-01T20:31:55Z) - NVIDIA FLARE: Federated Learning from Simulation to Real-World [11.490933081543787]
We created NVIDIA FLARE as an open-source development kit (SDK) to make it easier for data scientists to use FL in their research and real-world applications.
The SDK includes solutions for state-of-the-art FL algorithms and federated machine learning approaches.
arXiv Detail & Related papers (2022-10-24T14:30:50Z) - Federated Learning and Meta Learning: Approaches, Applications, and
Directions [94.68423258028285]
In this tutorial, we present a comprehensive review of FL, meta learning, and federated meta learning (FedMeta)
Unlike other tutorial papers, our objective is to explore how FL, meta learning, and FedMeta methodologies can be designed, optimized, and evolved, and their applications over wireless networks.
arXiv Detail & Related papers (2022-10-24T10:59:29Z) - FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in
Realistic Healthcare Settings [51.09574369310246]
Federated Learning (FL) is a novel approach enabling several clients holding sensitive data to collaboratively train machine learning models.
We propose a novel cross-silo dataset suite focused on healthcare, FLamby, to bridge the gap between theory and practice of cross-silo FL.
Our flexible and modular suite allows researchers to easily download datasets, reproduce results and re-use the different components for their research.
arXiv Detail & Related papers (2022-10-10T12:17:30Z) - FL_PyTorch: optimization research simulator for federated learning [1.6114012813668934]
Federated Learning (FL) has emerged as a promising technique for edge devices to collaboratively learn a shared machine learning model.
FL_PyTorch is a suite of open-source software written in python that builds on top of one the most popular research Deep Learning (DL) framework PyTorch.
arXiv Detail & Related papers (2022-02-07T12:18:28Z) - Federated Learning-based Active Authentication on Mobile Devices [98.23904302910022]
User active authentication on mobile devices aims to learn a model that can correctly recognize the enrolled user based on device sensor information.
We propose a novel user active authentication training, termed as Federated Active Authentication (FAA)
We show that existing FL/SL methods are suboptimal for FAA as they rely on the data to be distributed homogeneously.
arXiv Detail & Related papers (2021-04-14T22:59:08Z) - FedML: A Research Library and Benchmark for Federated Machine Learning [55.09054608875831]
Federated learning (FL) is a rapidly growing research field in machine learning.
Existing FL libraries cannot adequately support diverse algorithmic development.
We introduce FedML, an open research library and benchmark to facilitate FL algorithm development and fair performance comparison.
arXiv Detail & Related papers (2020-07-27T13:02:08Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.