Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers

• URL: http://arxiv.org/abs/2607.21074v1
• Date: Thu, 23 Jul 2026 09:07:56 GMT
• Title: Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers
• Abstract summary: Fine-tuning Vision Transformers (ViTs) with low-rank adapters (LoRA) promises better communication efficiency under federated setup.<n>Independently averaging LoRA factors is mathematically inconsistent, introducing cross-term aggregation error.<n>We propose SpecTraL, spectral transformation for layer-wise global rank discovery.
• Score: 0.9668407688201358
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning Vision Transformers (ViTs) with low-rank adapters (LoRA) promises better communication efficiency under federated setup, yet existing aggregation strategies face fundamental limitations. Independently averaging these LoRA factors is mathematically inconsistent, introducing cross-term aggregation error. In contrast, approaches that preserve heterogeneous client ranks by concatenating local adapters on the server substantially increase download cost and often require merging global LoRA updates into pretrained weights on the clients, causing reinitialization lag and unstable convergence. Other approaches further increase server-side overhead by reconstructing dense weight updates or training auxiliary models to refine aggregation error. In this work, we propose SpecTraL, spectral transformation for layer-wise global rank discovery, that resolves these challenges within a unified design. SpecTraL stacks local LoRA modules from clients and performs orthonormal Householder Transformation of the stacked adapters directly in the low-rank latent space, eliminating dense reconstruction of the global update and any auxiliary refinement on the server. By leveraging the Spiked Covariance Model from Random Matrix Theory, SpecTraL analytically separates the global consensus signal from non-IID noise, discovering optimal layer-wise global ranks without manual hyperparameter tuning. To match local ranks in subsequent rounds, we introduce a padding-aware initialization framework that lets clients incorporate residual LoRA dimensions without re-merging them into the pre-trained base model. Experiments on federated fine-tuning of ViT-B/16 and ViT-L/16 over DomainNet and NICO++ demonstrate improved accuracy-communication trade-offs, reduced server computation, and elimination of hyperparameter search for rank selection. Our code is available at https://github.com/DASS-Lab-Group/SpecTraL

Related papers

• SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks [15.20977008792474]
  SeFoRA is a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates.<n>SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model.<n>We prove convergence to a neighborhood of the first-order stationary point at rate $cO(1/T)$ for the rank-homogeneous setting.
  arXiv  Detail & Related papers  (2026-08-10T18:59:25Z)
• Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning [29.371418314402565]
  heterogeneous clients can make adapter aggregation unstable.<n>We propose Dynamic Subspace Boosting (Dysco) to allocate client-specific LoRA subspaces.<n>Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times.
  arXiv  Detail & Related papers  (2026-07-15T21:15:35Z)
• Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models [7.174865411448373]
  Federated fine-tuning of foundation models using Low-Rank Adaptation (LoRA) offers a communication efficient solution for distributed learning.<n>Existing federated LoRA methods suffer from two fundamental limitations: (1) structural aggregation bias, and (2) client side lag, as clients repeatedly re LoRA parameters across communication rounds, slowing convergence.<n>We propose HyperLoRA, a unified framework that addresses both issues through amortized federated adaptation through hypernetwork-driven LoRA generation and product space aggregation.
  arXiv  Detail & Related papers  (2026-06-04T13:28:48Z)
• FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation [70.67537583458572]
  Federated fine-tuning of foundation models with Low-Rank Adaptation (LoRA) provides efficient solution for reducing communication and computation costs.<n>The direct combination of FedAvg and LoRA suffers from three key issues: limited update space, inter-round state mismatch, and a client-agnostic starting state.<n>We propose FedSmoothLoRA, a federated LoRA tuning framework that preserves the enlarged update space, improves cross-round local optimization continuity, and provides a client-aware starting state.
  arXiv  Detail & Related papers  (2026-05-28T06:53:46Z)
• Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA [21.610540680175294]
  Federated LoRA enables parameter-efficient adaptation of large language models under decentralized data.<n> directly averaging LoRA factors is representation-dependent.<n>We propose textbfGLoRA, a gauge-aware server representation for federated LoRA.
  arXiv  Detail & Related papers  (2026-05-07T12:57:24Z)
• Improving Recursive Transformers with Mixture of LoRAs [2.672804414228544]
  Mixture of LoRAs (MoL) inserts Low-Rank Adaptation (LoRA) experts inside a shared feed-forward network (FFN)<n>MoL enables token-conditional weight-space modulation of the shared FFN without untying backbone parameters.<n>ModernALBERT achieves state-of-the-art performance among compact models.
  arXiv  Detail & Related papers  (2025-12-14T23:39:30Z)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape [52.98187034726091]
  We introduce Flat-LoRA, which aims to identify a low-rank adaptation situated in a flat region of the full parameter space.<n>We show that Flat-LoRA improves both in-domain and out-of-domain generalization.
  arXiv  Detail & Related papers  (2024-09-22T11:24:10Z)
• FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients [50.13097183691517]
  In real-world federated scenarios, there often exist a multitude of heterogeneous clients with varying computation and communication resources. We propose a novel federated tuning algorithm, FedRA. In each communication round, FedRA randomly generates an allocation matrix. It reorganizes a small number of layers from the original model based on the allocation matrix and fine-tunes using adapters.
  arXiv  Detail & Related papers  (2023-11-19T04:43:16Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Subspace based Federated Unlearning [75.90552823500633]
  Federated unlearning (FL) aims to remove a specified target client's contribution in FL to satisfy the user's right to be forgotten. Most existing federated unlearning algorithms require the server to store the history of the parameter updates. We propose a simple-yet-effective subspace based federated unlearning method, dubbed SFU, that lets the global model perform gradient ascent.
  arXiv  Detail & Related papers  (2023-02-24T04:29:44Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.