Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach

• URL: http://arxiv.org/abs/2608.09742v1
• Date: Mon, 10 Aug 2026 15:38:45 GMT
• Title: Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
• Abstract summary: We study whether $A$ should be shared across clients while $B$ remains client-specific (Share-A/Local-B) or whether $B$ should instead be shared while $A$ remains client-specific (Share-B/Local-A)<n>We propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance.
• Score: 58.97285859000893
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., $A$ and $B$, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether $A$ should be shared across clients while $B$ remains client-specific (Share-A/Local-B), or whether $B$ should instead be shared while $A$ remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-$r$ input-side space, whereas Share-B/Local-A requires a common rank-$r$ output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-$r$ input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.

Related papers

• SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks [15.20977008792474]
  SeFoRA is a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates.<n>SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model.<n>We prove convergence to a neighborhood of the first-order stationary point at rate $cO(1/T)$ for the rank-homogeneous setting.
  arXiv  Detail & Related papers  (2026-08-10T18:59:25Z)
• FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment [55.97027207627]
  Fed PDPO (Federated Personalized Direct Preference Optimization) is a personalized framework for preference alignment of large language models (LLMs)<n>It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation.<n>Experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.
  arXiv  Detail & Related papers  (2026-03-20T08:24:49Z)
• The Appeal and Reality of Recycling LoRAs with Adaptive Merging [33.0501514389443]
  We consider recycling LoRAs found "in the wild" on model repositories like the Hugging Face Hub.<n>We demonstrate that adaptive merging methods can improve performance over the base model but provide limited benefit over training a new LoRA.<n>This raises the possibility that adaptive merging from recycled LoRAs primarily works via some kind of regularization effect.
  arXiv  Detail & Related papers  (2026-02-12T18:59:20Z)
• Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs [26.212332132619736]
  We propose an asymmetric multi-LoRA design with multiple $A$ matrices and a single shared $B$ in multi-task fine-tuning.<n>Our methods achieve more balanced performance across tasks with comparable or superior average accuracy relative to existing multi-LoRA approaches.
  arXiv  Detail & Related papers  (2025-09-29T19:16:14Z)
• LoRAtorio: An intrinsic approach to LoRA Skill Composition [11.429106388558925]
  Low-Rank Adaptation (LoRA) has become a widely adopted technique in text-to-image diffusion models.<n>Existing approaches struggle to effectively compose multiple LoRA adapters.<n>We present LoRAtorio, a novel train-free framework for multi-LoRA composition.
  arXiv  Detail & Related papers  (2025-08-15T17:52:56Z)
• Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning [16.99490636203893]
  We present textscRavan, an adaptive multi-head LoRA method that balances parameter efficiency and model expressivity.<n>Experiments on vision and language benchmarks show that textscRavan improves test accuracy by 2-8% over prior parameter-efficient baselines.
  arXiv  Detail & Related papers  (2025-06-05T20:28:02Z)
• HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantization [55.972018549438964]
  Federated fine-tuning of pre-trained Large Language Models (LLMs) enables task-specific adaptation across diverse datasets while preserving privacy.<n>We propose HAFLQ (Heterogeneous Adaptive Federated Low-Rank Adaptation Fine-tuned LLM with Quantization), a novel framework for efficient and scalable fine-tuning of LLMs in heterogeneous environments.<n> Experimental results on the text classification task demonstrate that HAFLQ reduces memory usage by 31%, lowers communication cost by 49%, improves accuracy by 50%, and achieves faster convergence compared to the baseline method.
  arXiv  Detail & Related papers  (2024-11-10T19:59:54Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• Lifelong Personalized Low-Rank Adaptation of Large Language Models for Recommendation [50.837277466987345]
  We focus on the field of large language models (LLMs) for recommendation. We propose RecLoRA, which incorporates a Personalized LoRA module that maintains independent LoRAs for different users. We also design a Few2Many Learning Strategy, using a conventional recommendation model as a lens to magnify small training spaces to full spaces.
  arXiv  Detail & Related papers  (2024-08-07T04:20:28Z)
• Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning [57.36978335727009]
  Low-Rank Adaptation (LoRA) offers an efficient way to fine-tune large language models (LLMs) In this paper, we propose a framework that adaptively retrieves and composes multiple LoRAs based on input prompts.
  arXiv  Detail & Related papers  (2024-06-24T05:24:41Z)
• LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild [76.67343971195267]
  Low-Rank Adaptation (LoRA) provides an efficient solution for fine-tuning large language models (LLM) LoraRetriever is a retrieve-then-compose framework that adaptively retrieves and composes multiple LoRAs according to the input prompts. Experimental results indicate that LoraRetriever consistently outperforms the baselines.
  arXiv  Detail & Related papers  (2024-02-15T15:02:46Z)
• FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients [50.13097183691517]
  In real-world federated scenarios, there often exist a multitude of heterogeneous clients with varying computation and communication resources. We propose a novel federated tuning algorithm, FedRA. In each communication round, FedRA randomly generates an allocation matrix. It reorganizes a small number of layers from the original model based on the allocation matrix and fine-tunes using adapters.
  arXiv  Detail & Related papers  (2023-11-19T04:43:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.