A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine

• URL: http://arxiv.org/abs/2601.22124v1
• Date: Thu, 29 Jan 2026 18:48:21 GMT
• Title: A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine
• Authors: Anran Li, Yuanyuan Chen, Wenjun Long, Yu Yin, Yan Hu, Hyunjae Kim, Weipeng Zhou, Yujia Zhou, Hongyi Peng, Yang Ren, Xuguang Ai, Zhenyue Qin, Ming Hu, Xiaoxiao Li, Han Yu, Yih-Chung Tham, Lucila Ohno-Machado, Hua Xu, Qingyu Chen,
• Abstract summary: Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
• Score: 59.78991974851707
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis. To enable their use in clinical settings, LLMs are typically further adapted through continued pretraining or post-training using clinical data. However, most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems. Federated learning (FL) is a promising solution for enabling collaborative model development across healthcare institutions. Yet applying FL to LLMs in medicine remains fundamentally limited. First, conventional FL requires transmitting the full model during each communication round, which becomes impractical for multi-billion-parameter LLMs given the limited computational resources. Second, many FL algorithms implicitly assume data homogeneity, whereas real-world clinical data are highly heterogeneous across patients, diseases, and institutional practices. We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications. Fed-MedLoRA transmits only low-rank adapter parameters, reducing communication and computation overhead, while Fed-MedLoRA+ further incorporates adaptive, data-aware aggregation to improve convergence under cross-site heterogeneity. We apply the framework to clinical information extraction (IE), which transforms patient narratives into structured medical entities and relations. Accuracy was assessed across five patient cohorts through comparisons with BERT models, and LLaMA-3 and DeepSeek-R1, GPT-4o models. Evaluation settings included (1) in-domain training and testing, (2) external validation on independent cohorts, and (3) a low-resource new-site adaptation scenario using real-world clinical notes from the Yale New Haven Health System.

Related papers

• Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcare [8.87993480369896]
  Federated learning (FL) enables collaborative model training across decentralized medical institutions while preserving data privacy.<n>Med-MMFL is the first comprehensive MMFL benchmark for the medical domain, encompassing diverse modalities, tasks, and federation scenarios.<n>Our benchmark evaluates six representative state-of-the-art FL algorithms, covering different aggregation strategies, loss formulations, and regularization techniques.
  arXiv  Detail & Related papers  (2026-02-04T10:50:15Z)
• Counterfactual Modeling with Fine-Tuned LLMs for Health Intervention Design and Sensor Data Augmentation [2.6532805035238742]
  Counterfactual explanations (CFEs) provide human-centric interpretability by identifying the minimal, actionable changes required to alter a machine learning model's prediction.<n>We conduct a comprehensive evaluation of CF generation using large language models (LLMs)<n>We assess CFs across three dimensions: intervention quality, feature diversity, and augmentation effectiveness.
  arXiv  Detail & Related papers  (2026-01-21T02:04:08Z)
• BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text [14.409097921305134]
  BRIDGE is a comprehensive benchmark comprising 87 tasks sourced from real-world clinical data sources across nine languages.<n>It covers eight major task types spanning the entire continuum of patient care across six clinical stages and 20 representative applications.<n>Our results reveal substantial performance variation across model sizes, languages, natural language processing tasks, and clinical specialties.
  arXiv  Detail & Related papers  (2025-04-28T04:13:18Z)
• FedMetaMed: Federated Meta-Learning for Personalized Medication in Distributed Healthcare Systems [7.32609591220333]
  We introduce Federated Meta-Learning for Personalized Medication (FedMetaMed)<n>FedMetaMed combines federated learning and meta-learning to create models that adapt to diverse patient data across healthcare systems.<n>We show that FedMetaMed outperforms state-of-the-art FL methods, showing superior generalization even on out-of-the-art cohorts.
  arXiv  Detail & Related papers  (2024-12-05T03:36:55Z)
• FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data [52.55123685248105]
  Cardiovascular diseases (CVDs) are currently the leading cause of death worldwide, highlighting the critical need for early diagnosis and treatment. Machine learning (ML) methods can help diagnose CVDs early, but their performance relies on access to substantial data with high quality. This paper presents the first real-world FL benchmark for cardiovascular disease detection, named FedCVD.
  arXiv  Detail & Related papers  (2024-10-28T02:24:01Z)
• Communication-Efficient Hybrid Federated Learning for E-health with Horizontal and Vertical Data Partitioning [67.49221252724229]
  E-health allows smart devices and medical institutions to collaboratively collect patients' data, which is trained by Artificial Intelligence (AI) technologies to help doctors make diagnosis. Applying federated learning in e-health faces many challenges. Medical data is both horizontally and vertically partitioned. A naive combination of HFL and VFL has limitations including low training efficiency, unsound convergence analysis, and lack of parameter tuning strategies.
  arXiv  Detail & Related papers  (2024-04-15T19:45:07Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• Large Language Model Distilling Medication Recommendation Model [58.94186280631342]
  We harness the powerful semantic comprehension and input-agnostic characteristics of Large Language Models (LLMs)<n>Our research aims to transform existing medication recommendation methodologies using LLMs.<n>To mitigate this, we have developed a feature-level knowledge distillation technique, which transfers the LLM's proficiency to a more compact model.
  arXiv  Detail & Related papers  (2024-02-05T08:25:22Z)
• An In-Depth Evaluation of Federated Learning on Biomedical Natural Language Processing [7.412360079707614]
  Language models (LMs) have revolutionized natural language processing (NLP) Medical field faces challenges in training LMs due to limited data privacy constraints. In Federated Data (FL) we offer a decentralized solution that enables collaborative learning.
  arXiv  Detail & Related papers  (2023-07-20T22:10:04Z)
• Multi-Site Clinical Federated Learning using Recursive and Attentive Models and NVFlare [13.176351544342735]
  This paper develops an integrated framework that addresses data privacy and regulatory compliance challenges. It includes the development of an integrated framework that addresses data privacy and regulatory compliance challenges while maintaining elevated accuracy and substantiating the efficacy of the proposed approach.
  arXiv  Detail & Related papers  (2023-06-28T17:00:32Z)
• Auto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentation [48.821062916381685]
  Federated learning (FL) is a distributed machine learning technique that enables collaborative model training while avoiding explicit data sharing. In this work, we propose an efficient reinforcement learning(RL)-based federated hyperparameter optimization algorithm, termed Auto-FedRL. The effectiveness of the proposed method is validated on a heterogeneous data split of the CIFAR-10 dataset and two real-world medical image segmentation datasets.
  arXiv  Detail & Related papers  (2022-03-12T04:11:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.