Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment

• URL: http://arxiv.org/abs/2608.25114v1
• Date: Tue, 25 Aug 2026 20:09:44 GMT
• Title: Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment
• Abstract summary: We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications.<n>We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation.
• Score: 51.83961778298564
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) has emerged as a key approach for training models across decentralized data, yet benchmarking in FL remains difficult to reproduce, compare, and extend. Existing evaluations are often tied to custom infrastructure, released as incomplete research code, and conducted primarily in simulation, which limits portability and practical relevance. We present Flower Hub, a platform for publishing, discovering, and executing decentralized and federated applications. We show how it enables reproducible benchmarking by packaging benchmarks as executable, versioned applications with standardized metadata, pinned dependencies, and explicit evaluation workflows. We instantiate this approach with a multi-domain benchmark suite spanning cross-silo and cross-device settings, and including tasks in medical imaging, financial tabular learning, legal instruction tuning, phishing URL detection, and audio tagging. We further demonstrate that the same benchmarking application can run across both simulation and deployment runtimes without changing the application code, enabling unified evaluation across varying learning environments. Beyond model quality, our benchmark design supports system-aware reporting, including runtime and communication metrics. This work advances benchmarking in FL settings from ad hoc code artifacts towards portable, executable, and reusable benchmark applications.

Related papers

• UNIFERENCE: A Discrete Event Simulation Framework for Developing Distributed AI Models [0.42015796373418257]
  UNI is a discrete-event simulation framework for developing, benchmarking, and deploying distributed AI models.<n>It integrates seamlessly with PyTorch Distributed, enabling the same transition from simulation to real deployment.
  arXiv  Detail & Related papers  (2026-03-27T14:35:02Z)
• UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities [70.79422099851506]
  We present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems.<n>UDE standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface.
  arXiv  Detail & Related papers  (2026-03-24T13:01:31Z)
• DEP: A Decentralized Large Language Model Evaluation Protocol [51.3646001384887]
  Decentralized Evaluation Protocol (DEP) is a decentralized yet unified and standardized evaluation framework.<n>By decoupling users, LLMs, and benchmarks, DEP enables modular, plug-and-play evaluation.<n>We develop DEP Toolkit, a protocol-compatible toolkit that supports features such as breakpoint resume, concurrent requests, and congestion control.
  arXiv  Detail & Related papers  (2026-03-01T16:10:16Z)
• Easy Data Unlearning Bench [53.1304932656586]
  We introduce a unified and benchmarking suite that simplifies the evaluation of unlearning algorithms.<n>By standardizing setup and metrics, it enables reproducible, scalable, and fair comparison across unlearning methods.
  arXiv  Detail & Related papers  (2026-02-18T12:20:32Z)
• FLsim: A Modular and Library-Agnostic Simulation Framework for Federated Learning [3.62218729239779]
  Federated Learning (FL) has undergone significant development since its inception in 2016.<n>We introduce FLsim, a comprehensive FL simulation framework designed to meet the diverse requirements of FL in the literature.<n>We demonstrate the effectiveness and versatility of FLsim in a diverse range of state-of-the-art FL experiments.
  arXiv  Detail & Related papers  (2025-07-15T15:53:01Z)
• Benchmarking Federated Learning for Semantic Datasets: Federated Scene Graph Generation [3.499870393443268]
  Federated learning (FL) enables decentralized training while preserving data privacy, yet existing FL benchmarks address relatively simple classification tasks.<n>We propose a benchmark process to establish an FL benchmark with controllable semantic heterogeneity across clients.<n>As a proof of concept, we construct a federated PSG benchmark, demonstrating the efficacy of the existing PSG methods in an FL setting.
  arXiv  Detail & Related papers  (2024-12-11T08:10:46Z)
• SeBS-Flow: Benchmarking Serverless Cloud Function Workflows [51.4200085836966]
  We propose the first serverless workflow benchmarking suite SeBS-Flow.<n>SeBS-Flow includes six real-world application benchmarks and four microbenchmarks representing different computational patterns.<n>We conduct comprehensive evaluations on three major cloud platforms, assessing performance, cost, scalability, and runtime deviations.
  arXiv  Detail & Related papers  (2024-10-04T14:52:18Z)
• FedModule: A Modular Federated Learning Framework [5.872098693249397]
  Federated learning (FL) has been widely adopted across various applications, such as healthcare, finance, and smart cities. This paper introduces FedModule, a flexible and FL experimental framework. FedModule adheres to the "one code, all scenarios" principle and employs a modular design that breaks the FL process into individual components.
  arXiv  Detail & Related papers  (2024-09-07T15:03:12Z)
• FS-Real: Towards Real-World Cross-Device Federated Learning [60.91678132132229]
  Federated Learning (FL) aims to train high-quality models in collaboration with distributed clients while not uploading their local data. There is still a considerable gap between the flourishing FL research and real-world scenarios, mainly caused by the characteristics of heterogeneous devices and its scales. We propose an efficient and scalable prototyping system for real-world cross-device FL, FS-Real.
  arXiv  Detail & Related papers  (2023-03-23T15:37:17Z)
• FedScale: Benchmarking Model and System Performance of Federated Learning [4.1617240682257925]
  FedScale is a set of challenging and realistic benchmark datasets for federated learning (FL) research. FedScale is open-source with permissive licenses and actively maintained.
  arXiv  Detail & Related papers  (2021-05-24T15:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.