Subspace-Constrained Federated Learning with Low-Rank Adaptation

• URL: http://arxiv.org/abs/2606.22724v1
• Date: Sun, 21 Jun 2026 23:49:22 GMT
• Title: Subspace-Constrained Federated Learning with Low-Rank Adaptation
• Abstract summary: Federated low-rank adaptation methods are attractive for fine-tuning large models under communication and privacy constraints.<n>We study whether this subspace misalignment leads to destructive aggregation and slower convergence in LoRA-based federated learning.<n>We propose a subspace-regularized federated LoRA objective that encourages local client updates to remain close to a shared global reference subspace.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated low-rank adaptation methods are attractive for fine-tuning large models under communication and privacy constraints, but heterogeneous client data can induce geometric misalignment between local low-rank updates. We study whether this subspace misalignment leads to destructive aggregation and slower convergence in LoRA-based federated learning. We propose a subspace-regularized federated LoRA objective that encourages local client updates to remain close to a shared global reference subspace. We present a complete empirical evaluation on two pretrained models, RoBERTa-large and SmolLM-360M, over HellaSwag in a non-IID 10-client federated setting, across 3 random seeds (42, 43, 44), yielding 24 total experimental runs (4 methods x 3 seeds x 2 models). On RoBERTa-large, Subspace-Reg achieves the strongest mean best accuracy (0.454 +/- 0.023), mean final accuracy (0.429 +/- 0.011), and lowest final loss (1.363) across all three seeds, outperforming FedAvg, SVD redistribution, and FedSVD baselines by a large margin. On SmolLM-360M, FedAvg leads on accuracy, revealing that accuracy gains are model-dependent. Crucially, Subspace-Reg achieves near-perfect basis overlap, approximately 0.9999, on both models and across all seeds, versus 0.958 to 0.991 for all baselines, providing robust support for the geometric alignment hypothesis. The code is publicly available at https://github.com/sadia-sigma-lab/Subspace-Constrained-Federated-learning-with-Lora.

Related papers

• Solution for UCF UrbanTwin LUMPI Track: Sim-to-Real Urban LiDAR 3D Object Detection [54.03436243843709]
  We present our solution to the LUMPI track of the UCF UrbanTwin Sim2Real LiDAR Challenge at the 6th DriveX Workshop, ECCV 2026.<n>The detector must be trained only on synthetic data and is evaluated on 50 held-out real LiDAR frames.<n>A separate 50-frame synthetic submission is evaluated for point-cloud realism.
  arXiv  Detail & Related papers  (2026-09-07T15:03:36Z)
• SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks [15.20977008792474]
  SeFoRA is a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates.<n>SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model.<n>We prove convergence to a neighborhood of the first-order stationary point at rate $cO(1/T)$ for the rank-homogeneous setting.
  arXiv  Detail & Related papers  (2026-08-10T18:59:25Z)
• Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning [29.371418314402565]
  heterogeneous clients can make adapter aggregation unstable.<n>We propose Dynamic Subspace Boosting (Dysco) to allocate client-specific LoRA subspaces.<n>Dysco substantially reduces interference, reduces the final-round synthetic training loss by up to 9 times.
  arXiv  Detail & Related papers  (2026-07-15T21:15:35Z)
• Beyond Inference-Only Deployment: Comparing Weight-Based Consolidation Against Cascading Compaction [1.6214121783846343]
  We evaluate a nightly consolidation of interaction knowledge into model weights via synthesis, synthesis, and Low-Rank Adaptation (LoRA) fine-tuning on a single consumer.<n>Across ten realistic software development conversations, cascading compaction retain 36.8 +/- 3.0% of knowledge.<n>LoRA consolidates interaction knowledge into model weights via synthesis, synthesis, and Low-Rank Adaptation (LoRA) fine-tuning on a single consumer.
  arXiv  Detail & Related papers  (2026-05-23T16:57:25Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• When More Parameters Hurt: Foundation Model Priors Amplify Worst-Client Disparity Under Extreme Federated Heterogeneity [0.0]
  Federated learning (FL) is increasingly used to fine-tune foundation models (FMs) on distributed private data.<n>We compare worst-client accuracy between TextCNN (2.7M parameters) and DistilBERT with Low-Rank Adaptation (LoRA)<n>Under extreme label skew (alpha = 0.1), DistilBERT+LoRA produces a worst-client accuracy gap of 50.1% -- 56% larger than TextCNN's 32.2% gap, despite having 25x more parameters and extensive pretraining.
  arXiv  Detail & Related papers  (2026-05-09T15:22:08Z)
• Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning [6.087223232267447]
  Differentially private federated fine-tuning of large models with LoRA suffers from aggregation error caused by LoRA's multiplicative structure.<n>We propose AS-LoRA, an adaptive framework defined by three axes (i.e. layer-wise freedom, round-wise adaptivity, and curvature-aware score)<n>Theoretically, AS-LoRA eliminates the reconstruction-error floor of layer-tied schedules, accelerates convergence, implicitly biases solutions toward flatter minima, and incurs no additional privacy cost.
  arXiv  Detail & Related papers  (2026-05-07T07:01:00Z)
• Deterministic Fuzzy Triage for Legal Compliance Classification and Evidence Retrieval [0.0]
  Legal teams increasingly use machine learning to triage large volumes of contractual evidence.<n>Many models are opaque, non-deterministic, and difficult to align with frameworks such as HIPAA or NERC-CIP.<n>We study a simple, reproducible alternative based on deterministic dual encoders and transparent fuzzy triage bands.
  arXiv  Detail & Related papers  (2026-03-08T00:31:34Z)
• Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models [14.755143405057929]
  Fine-tuning large vision models (LVMs) and large language models (LLMs) under differentially private learning (DPFL) is hindered by a fundamental privacy-utility trade-off.<n>Low-Rank Adaptation (LoRA), a promising parameter-efficient fine-tuning (PEFT) method, reduces computational and communication costs by introducing two trainable low-rank matrices while freezing pre-trained weights.<n>We propose LA-LoRA, a novel approach that decouples gradient interactions and aligns update directions across clients to enhance robustness under stringent privacy constraints.
  arXiv  Detail & Related papers  (2026-02-23T15:05:28Z)
• Perfectly-Private Analog Secure Aggregation in Federated Learning [51.61616734974475]
  In federated learning, multiple parties train models locally and share their parameters with a central server, which aggregates them to update a global model.<n>In this paper, a novel secure parameter aggregation method is proposed that employs the torus rather than a finite field.
  arXiv  Detail & Related papers  (2025-09-10T15:22:40Z)
• Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping [64.58402571292723]
  We propose a manifold reshaping approach called FedMR to calibrate the feature space of local training. We conduct extensive experiments on a range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency.
  arXiv  Detail & Related papers  (2024-05-29T10:56:13Z)
• DFedADMM: Dual Constraints Controlled Model Inconsistency for Decentralized Federated Learning [52.83811558753284]
  Decentralized learning (DFL) discards the central server and establishes a decentralized communication network. Existing DFL methods still suffer from two major challenges: local inconsistency and local overfitting.
  arXiv  Detail & Related papers  (2023-08-16T11:22:36Z)
• $\texttt{FedBC}$: Calibrating Global and Local Models via Federated Learning Beyond Consensus [66.62731854746856]
  In federated learning (FL), the objective of collaboratively learning a global model through aggregation of model updates across devices tends to oppose the goal of personalization via local information. In this work, we calibrate this tradeoff in a quantitative manner through a multi-criterion-based optimization. We demonstrate that $texttFedBC$ balances the global and local model test accuracy metrics across a suite datasets.
  arXiv  Detail & Related papers  (2022-06-22T02:42:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.