Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning

• URL: http://arxiv.org/abs/2509.15087v1
• Date: Thu, 18 Sep 2025 15:43:33 GMT
• Title: Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
• Authors: Lei Wang, Jieming Bian, Letian Zhang, Jie Xu,
• Abstract summary: Federated Learning (FL) offers a privacy-preserving solution, but faces challenges with computational constraints.<n>Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient fine-tuning approach.<n>We propose FedLEASE, a novel framework that adaptively clusters clients based on representation similarity to allocate and train domain-specific LoRA experts.
• Score: 12.733972494875713
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities across various tasks, but fine-tuning them for domain-specific applications often requires substantial domain-specific data that may be distributed across multiple organizations. Federated Learning (FL) offers a privacy-preserving solution, but faces challenges with computational constraints when applied to LLMs. Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient fine-tuning approach, though a single LoRA module often struggles with heterogeneous data across diverse domains. This paper addresses two critical challenges in federated LoRA fine-tuning: 1. determining the optimal number and allocation of LoRA experts across heterogeneous clients, and 2. enabling clients to selectively utilize these experts based on their specific data characteristics. We propose FedLEASE (Federated adaptive LoRA Expert Allocation and SElection), a novel framework that adaptively clusters clients based on representation similarity to allocate and train domain-specific LoRA experts. It also introduces an adaptive top-$M$ Mixture-of-Experts mechanism that allows each client to select the optimal number of utilized experts. Our extensive experiments on diverse benchmark datasets demonstrate that FedLEASE significantly outperforms existing federated fine-tuning approaches in heterogeneous client settings while maintaining communication efficiency.

Related papers

• CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging [49.87105462292961]
  Core Space Mixture of LoRA (bfCoMoL) is a novel MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation.<n>CoMoL consistently outperforms existing methods across multiple tasks.
  arXiv  Detail & Related papers  (2026-02-28T09:40:11Z)
• FeDecider: An LLM-Based Framework for Federated Cross-Domain Recommendation [75.50721642765994]
  Large language model (LLM)-based recommendation models have demonstrated impressive performance.<n>We propose an LLM-based framework for Federated cross-domain recommendation, FeDecider.<n>Extensive experiments across diverse datasets validate the effectiveness of our proposed FeDecider.
  arXiv  Detail & Related papers  (2026-02-17T21:42:28Z)
• Semantic-guided LoRA Parameters Generation [22.648880814012184]
  Low-Rank Adaptation (LoRA) has demonstrated strong generalization capabilities across a variety of tasks for efficiently fine-tuning AI models.<n>SG-LoRA is the first of its kind framework to efficiently produce user-specific LoRA without additional training on user tasks or access to user-specific data.<n>SG-LoRA enables the real-time construction of LoRA models aligned with individual intents by distilling knowledge from prominent LoRA experts.
  arXiv  Detail & Related papers  (2025-09-05T14:43:41Z)
• FediLoRA: Heterogeneous LoRA for Federated Multimodal Fine-tuning under Missing Modalities [9.507134068207597]
  FediLoRA is a framework for federated multimodal fine-tuning under heterogeneous LoRA ranks and missing modalities.<n>It achieves superior performance over competitive baselines in both global and personalized settings.
  arXiv  Detail & Related papers  (2025-09-01T10:40:13Z)
• Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMs [37.03583502049329]
  Fine-tuning large language models (LLMs) on resource-constrained clients remains a challenging problem.<n>Recent works have fused low-rank adaptation (LoRA) techniques with federated fine-tuning to mitigate challenges associated with client model sizes and data scarcity.<n>We propose federated sketching LoRA, which leverages a sketching mechanism to enable clients to update submatrices of global LoRA modules maintained by the server.
  arXiv  Detail & Related papers  (2025-01-31T18:44:35Z)
• Client-Centric Federated Adaptive Optimization [78.30827455292827]
  Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private.<n>We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
  arXiv  Detail & Related papers  (2025-01-17T04:00:50Z)
• Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions [59.5243730853157]
  Federated learning (FL) provides a privacy-preserving solution for fine-tuning pre-trained large language models (LLMs) using distributed private datasets.<n>This article conducts a comparative analysis of three advanced federated LLM (FedLLM) frameworks that integrate knowledge distillation (KD) and split learning (SL) to mitigate these issues.
  arXiv  Detail & Related papers  (2025-01-08T11:37:06Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures [15.645254436094055]
  Federated Learning (FL) enables collaborative fine-tuning of Large Language Models without accessing raw data.<n>We propose FedAMoLE, a lightweight personalized FL framework that enables data-driven heterogeneous model architectures.<n> Experiments show that FedAMoLE improves client-side performance by an average of 5.14% compared to existing approaches.
  arXiv  Detail & Related papers  (2024-11-28T13:20:38Z)
• Fed-pilot: Optimizing LoRA Allocation for Efficient Federated Fine-Tuning with Heterogeneous Clients [11.102441622530181]
  We propose Fed-pilot, a memory-efficient federated fine-tuning framework.<n>It enables memory-constrained clients to participate in Low-Rank Adaptation (LoRA)-based fine-tuning by training only a subset of LoRA modules locally.<n>To the best of our knowledge, this is the first study on federated fine-tuning of FMs that integrates memory-constrained optimization.
  arXiv  Detail & Related papers  (2024-10-14T06:36:41Z)
• Optimizing Cross-Client Domain Coverage for Federated Instruction Tuning of Large Language Models [87.49293964617128]
  Federated domain-specific instruction tuning (FedDIT) for large language models (LLMs) aims to enhance performance in specialized domains using distributed private and limited data.<n>We empirically establish that cross-client domain coverage, rather than data heterogeneity, is the pivotal factor.<n>We introduce FedDCA, an algorithm that explicitly maximizes this coverage through diversity-oriented client center selection and retrieval-based augmentation.
  arXiv  Detail & Related papers  (2024-09-30T09:34:31Z)
• Lifelong Personalized Low-Rank Adaptation of Large Language Models for Recommendation [50.837277466987345]
  We focus on the field of large language models (LLMs) for recommendation. We propose RecLoRA, which incorporates a Personalized LoRA module that maintains independent LoRAs for different users. We also design a Few2Many Learning Strategy, using a conventional recommendation model as a lens to magnify small training spaces to full spaces.
  arXiv  Detail & Related papers  (2024-08-07T04:20:28Z)
• Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources [31.041608465716575]
  Federated Learning (FL) has recently been applied to the parameter-efficient fine-tuning of Large Language Models (LLMs) This study introduces FlexLoRA, a simple yet effective aggregation scheme for LLM fine-tuning.
  arXiv  Detail & Related papers  (2024-02-18T08:32:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.