Benchmark Evaluation of Feredated Learning on Multi-organ Images

• URL: http://arxiv.org/abs/2607.08219v1
• Date: Thu, 09 Jul 2026 08:14:15 GMT
• Title: Benchmark Evaluation of Feredated Learning on Multi-organ Images
• Abstract summary: MobenFL is a benchmark that integrates cutting-edge FL algorithms and 22 medical imaging datasets.<n>It provides a comprehensive and in-depth evaluation framework for the clinical application of FL in the medical field.
• Score: 4.5759207062782234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The privacy requirements of medical data and its substantial variations across organs and modalities hinder the clinical implementation of medical AI. Federated learning (FL) is a feasible approach to overcome these challenges. Due to the continuous emergence of FL algorithms and the highly heterogeneous nature of medical data, objectively evaluating their performance in real-world clinical settings remains difficult. Therefore, a comprehensive federated medical imaging benchmark, serving as a unified evaluation standard, is crucial for advancing the technology toward reliable clinical application. Existing federated medical imaging benchmarks have not yet adequately incorporated state-of-the-art algorithms, are limited to data from single organs or modalities, and overly emphasize model accuracy, making it difficult to comprehensively assess the overall efficacy of FL in real-world medical environments. To address these challenges, we developed the MobenFL benchmark. This benchmark integrates 20 cutting-edge FL algorithms and 22 medical imaging datasets, covering 12 critical organs across the human body, surpassing existing benchmark in breadth. In terms of evaluation dimensions, MobenFL not only assesses performance but also systematically incorporates key metrics such as algorithmic efficiency and privacy protection capabilities. Additionally, it conducts specialized evaluations for complex real-world clinical scenarios involving different diseases, devices, and imaging modalities, thereby providing a comprehensive and in-depth evaluation framework for the clinical application of FL in the medical field.

Related papers

• QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models [8.050059911826338]
  Large Language Models (LLMs) excel on standardized medical exams, but high scores often fail to translate to high-quality responses for real-world medical queries.<n>We introduce QuarkMedBench, an ecologically valid benchmark tailored for real-world medical LLM assessment.
  arXiv  Detail & Related papers  (2026-03-14T01:51:43Z)
• FL-MedSegBench: A Comprehensive Benchmark for Federated Learning on Medical Image Segmentation [57.930067699003764]
  Federated learning (FL) offers a privacy-preserving paradigm for collaborative medical image analysis without sharing raw data.<n>We introduce FL-MedSegBench, the first comprehensive benchmark for federated learning on medical image segmentation.<n>We systematically evaluate eight generic FL (gFL) and five personalized FL (pFL) methods across multiple dimensions.
  arXiv  Detail & Related papers  (2026-03-12T08:28:41Z)
• Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcare [8.87993480369896]
  Federated learning (FL) enables collaborative model training across decentralized medical institutions while preserving data privacy.<n>Med-MMFL is the first comprehensive MMFL benchmark for the medical domain, encompassing diverse modalities, tasks, and federation scenarios.<n>Our benchmark evaluates six representative state-of-the-art FL algorithms, covering different aggregation strategies, loss formulations, and regularization techniques.
  arXiv  Detail & Related papers  (2026-02-04T10:50:15Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• Medical Imaging AI Competitions Lack Fairness [50.895929923643905]
  We assess fairness along two complementary dimensions: whether challenge datasets are representative of real-world clinical diversity, and whether they are accessible and legally reusable in line with the FAIR principles.<n>Our findings show substantial biases in dataset composition, including geographic location, modality, and problem type-related biases, indicating that current benchmarks do not adequately reflect real-world clinical diversity.<n>These shortcomings expose foundational limitations in our benchmarking ecosystem and highlight a disconnect between leaderboard success and clinical relevance.
  arXiv  Detail & Related papers  (2025-12-19T13:48:10Z)
• FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data [52.55123685248105]
  Cardiovascular diseases (CVDs) are currently the leading cause of death worldwide, highlighting the critical need for early diagnosis and treatment. Machine learning (ML) methods can help diagnose CVDs early, but their performance relies on access to substantial data with high quality. This paper presents the first real-world FL benchmark for cardiovascular disease detection, named FedCVD.
  arXiv  Detail & Related papers  (2024-10-28T02:24:01Z)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI [67.09501109871351]
  Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals. GMAI-MMBench is the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format.
  arXiv  Detail & Related papers  (2024-08-06T17:59:21Z)
• On the Impact of Data Heterogeneity in Federated Learning Environments with Application to Healthcare Networks [3.9058850780464884]
  Federated Learning (FL) allows privacy-sensitive applications to leverage their dataset for a global model construction without any disclosure of the information. One of those domains is healthcare, where groups of silos collaborate in order to generate a global predictor with improved accuracy and generalization. This paper presents a comprehensive exploration of the mathematical formalization and taxonomy of heterogeneity within FL environments, focusing on the intricacies of medical data.
  arXiv  Detail & Related papers  (2024-04-29T09:05:01Z)
• MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation [110.31526448744096]
  We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. We are building MedPerf, an open framework for benchmarking machine learning in the medical domain.
  arXiv  Detail & Related papers  (2021-09-29T18:09:41Z)
• The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation [81.72197368690031]
  We present a new benchmarking suite designed specifically for medical sequential decision making. The Medkit-Learn(ing) Environment is a publicly available Python package providing simple and easy access to high-fidelity synthetic medical data.
  arXiv  Detail & Related papers  (2021-06-08T10:38:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.