Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

• URL: http://arxiv.org/abs/2607.14367v1
• Date: Wed, 15 Jul 2026 21:15:35 GMT
• Title: Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning
• Abstract summary: heterogeneous clients can make adapter aggregation unstable.<n>We propose Dynamic Subspace Boosting (Dysco) to allocate client-specific LoRA subspaces.<n>Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times.
• Score: 29.371418314402565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated fine-tuning of large pre-trained models increasingly relies on Low-Rank Adaptation (LoRA) to reduce communication and computation, but heterogeneous clients can make adapter aggregation unstable. We identify the data-parameter interference as a geometric source of this instability. This interference is controlled by the alignment between LoRA update subspaces and client activations, suggesting that federated LoRA aggregation should be viewed not only as parameter averaging but also as subspace allocation. We propose Dynamic Subspace Boosting (Dysco), a plug-in method that allocates client-specific LoRA subspaces in a federated and dynamic manner. In each round, clients compute activation-insensitive subspaces from local representations and transmit only the resulting bases; the server then constructs client-specific merged subspaces through a closed-form solution that maximizes compatibility with other clients' insensitive directions. To handle representation drift, Dysco performs multi-round subspace boosting to preserve past update directions while adapting to future representations. We provide a convergence analysis that embeds the data-parameter interference as an aggregation-error term in a standard federated optimization bound, and prove that Dysco's server-fixed merged subspaces yield a tighter upper bound on this error. Experiments on controlled synthetic federated tasks and on MIMIC-IV clinical-note classification with Llama-3.2-1B show that Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times relative to baselines under the orthogonal-subspace partition the theory identifies, improves all five tested FL algorithms by up to 4.3% on MIMIC, outperforms recent federated LoRA methods, and adds only 0.9% wall-clock overhead. Our code is available at https://github.com/illidanlab/Dysco.

Related papers

• SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks [15.20977008792474]
  SeFoRA is a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates.<n>SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model.<n>We prove convergence to a neighborhood of the first-order stationary point at rate $cO(1/T)$ for the rank-homogeneous setting.
  arXiv  Detail & Related papers  (2026-08-10T18:59:25Z)
• Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers [0.9668407688201358]
  Fine-tuning Vision Transformers (ViTs) with low-rank adapters (LoRA) promises better communication efficiency under federated setup.<n>Independently averaging LoRA factors is mathematically inconsistent, introducing cross-term aggregation error.<n>We propose SpecTraL, spectral transformation for layer-wise global rank discovery.
  arXiv  Detail & Related papers  (2026-07-23T09:07:56Z)
• SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling [17.873016968298483]
  Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.<n>We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that re parameterizes a rank-rtot update as a sum of K static experts.
  arXiv  Detail & Related papers  (2026-06-27T03:25:36Z)
• SSR-Merge: Subspace Signal Routing for Training-Free LoRA Merging in Diffusion Models [27.413287599101327]
  Low-Rank Adaptation (LoRA) merging can efficiently combine diverse generative capabilities from multiple trained LoRAs for a diffusion model.<n>Existing LoRA merging techniques often suffer from severe parameter interference, causing destructive collisions in the shared parameter space.<n>We propose Subspace Signal Routing (SSR), which resolves interference by routing internal signals instead of performing parameter-space merge.
  arXiv  Detail & Related papers  (2026-06-09T09:16:17Z)
• FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation [70.67537583458572]
  Federated fine-tuning of foundation models with Low-Rank Adaptation (LoRA) provides efficient solution for reducing communication and computation costs.<n>The direct combination of FedAvg and LoRA suffers from three key issues: limited update space, inter-round state mismatch, and a client-agnostic starting state.<n>We propose FedSmoothLoRA, a federated LoRA tuning framework that preserves the enlarged update space, improves cross-round local optimization continuity, and provides a client-aware starting state.
  arXiv  Detail & Related papers  (2026-05-28T06:53:46Z)
• Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment [4.733680065352959]
  Low-rank adaptation (LoRA) has emerged as a powerful tool for parameter-efficient fine-tuning of large language models (LLMs)<n>This paper studies LoRA under a federated learning setting, enabling collaborative fine-tuning across clients while preserving parameter efficiency.<n>We propose Collaborative Low-rank Alignment and Identifiable Recovery (CLAIR), a contamination-aware framework that relies only on preliminary local estimators.
  arXiv  Detail & Related papers  (2026-05-20T14:12:16Z)
• FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA [25.49850401602623]
  Federated LoRA provides a communication-efficient mechanism for fine-tuning large language models on decentralized data.<n>In practice, a discrepancy between the factor-wise averaging used to preserve low rank and the mathematically correct aggregation of local updates can cause significant aggregation error and unstable training.<n>We propose FedRot-LoRA, a framework that aligns client updates via transformations prior to aggregation.
  arXiv  Detail & Related papers  (2026-02-27T03:18:32Z)
• Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA [61.12136997430116]
  Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model (LLM) by exchanging locally updated parameters with a subset of neighboring devices via wireless connections for knowledge integration.<n> directly aggregating parameters fine-tuned on heterogeneous datasets induces three primary issues across the DFL life-cycle: (i) catastrophic knowledge forgetting during fine-tuning process, arising from conflicting update directions caused by data heterogeneity; (ii) textitinefficient communication and convergence during model aggregation process,
  arXiv  Detail & Related papers  (2026-02-24T02:45:32Z)
• Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment [45.726776332713136]
  We propose FedGaLore, which combines client-side GaLore-style gradient-subspace optimization with server-side drift-robust synchronization of projected second-moment states.<n>Across NLU, vision, and NLG benchmarks, FedGaLore improves robustness and accuracy over state-of-the-art federated LoRA baselines in non-IID settings.
  arXiv  Detail & Related papers  (2026-02-02T07:27:44Z)
• Regularizing Subspace Redundancy of Low-Rank Adaptation [54.473090597164834]
  We propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation.<n>Our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks.<n>As a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs.
  arXiv  Detail & Related papers  (2025-07-28T11:52:56Z)
• DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptation [29.30782543513243]
  We propose a privacy-preserving federated finetuning framework called underlineDeviation underlineEliminating and Noisunderlinee underlineRegulating (DEeR) We show that DEeR shows better performance on public medical datasets in comparison with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-10-16T18:11:52Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Subspace based Federated Unlearning [75.90552823500633]
  Federated unlearning (FL) aims to remove a specified target client's contribution in FL to satisfy the user's right to be forgotten. Most existing federated unlearning algorithms require the server to store the history of the parameter updates. We propose a simple-yet-effective subspace based federated unlearning method, dubbed SFU, that lets the global model perform gradient ascent.
  arXiv  Detail & Related papers  (2023-02-24T04:29:44Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.