FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA

• URL: http://arxiv.org/abs/2608.03267v1
• Date: Tue, 04 Aug 2026 07:39:40 GMT
• Title: FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA
• Abstract summary: We propose a geometry-consistent aggregation framework for differentially private federated LoRA.<n>FedGSA represents each privatized client update as a basis-invariant subspace on the Grassmann manifold.<n>We prove that FedGSA incurs no additional privacy loss beyond client-side DP training and establish its convergence under standard assumptions.
• Score: 15.652808783124835
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Low-Rank Adaptation (LoRA) enables communication-efficient federated fine-tuning of pretrained language models. However, integrating differential privacy (DP) into federated LoRA remains challenging: independently perturbing and aggregating its two low-rank matrices can cause aggregation mismatch and the quadratic noise term. Existing methods mitigate these issues by freezing one low-rank matrix but still rely on Euclidean aggregation, which is basis-dependent and may distort the global update. To address this limitation, we propose FedGSA, a geometry-consistent aggregation framework for differentially private federated LoRA. FedGSA represents each privatized client update as a basis-invariant subspace on the Grassmann manifold. In each communication round, clients extract low-dimensional subspaces capturing dominant update directions and encode them as projection matrices. The server aggregates these representations to estimate a geometry-consistent global update subspace and reconstructs the global LoRA factors within it, reducing distortion caused by basis misalignment, privacy noise, and heterogeneous client updates. We prove that FedGSA incurs no additional privacy loss beyond client-side DP training and establish its convergence under standard assumptions. Experiments on four GLUE tasks and a language generation benchmark demonstrate consistent improvements across privacy budgets and degrees of data heterogeneity. In particular, FedGSA improves average accuracy over the strongest baseline by 2.17% and 2.27% under $ε=6$ and $ε=3$, respectively.

Related papers

• Privacy-Preserving Topology-Guided Safety for LLM-Based Multi-Agent Systems via Federated Graph Learning [67.04448659688579]
  FGLGuard cuts AgentDojo's ground-success rate by 43% at near-unguarded utility, zero API cost, and negligible capability loss.<n>Federation is not optional: off-the-shelf transfer collapses under distribution shift.<n>Live FGLGuard cuts AgentDojo's ground-success rate by 43% at near-unguarded utility, zero API cost, and negligible capability loss.
  arXiv  Detail & Related papers  (2026-09-02T07:57:12Z)
• Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity [12.733972494875713]
  FedRoRA is a novel framework that enables fine-grained personalization within rank-heterogeneous federations.<n>It decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales.<n>Experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-09-01T03:09:20Z)
• FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA [3.3855343208256556]
  Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors.<n>We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks.
  arXiv  Detail & Related papers  (2026-08-15T19:20:31Z)
• Improving Parameter-Efficient Federated Learning with Differentially Private Refactorization [8.171217925484921]
  Federated Learning (FL) with parameter-efficient fine-tuning enables scalable model training on distributed data.<n>Low-Rank Adaptation (LoRA) often introduces errors during global aggregation and amplifies the negative effect of Differential Privacy (DP)<n>We propose FedPower, a differentially private cross-silo FL framework that reshapes server-side aggregation.
  arXiv  Detail & Related papers  (2026-05-08T20:06:09Z)
• UMEDA: Unified Multi-modal Efficient Data Fusion for Privacy-Preserving Graph Federated Learning via Spectral-Gated Attention and Diffusion-Based Operator Alignment [20.399005485708273]
  Device-free localization trains models from heterogeneous wireless and visual sensors distributed across edge devices.<n>We propose UMEDA, a graph federated learning framework in which clients form nodes of a global graph that share a continuous integral operator.<n> UMEDA outperforms state-of-the-art federated baselines in accuracy, convergence, and communication efficiency.
  arXiv  Detail & Related papers  (2026-05-08T08:15:47Z)
• Byzantine-Robust and Differentially Private Federated Optimization under Weaker Assumptions [89.52532304099522]
  Federated Learning (FL) enables clients to collaboratively train a shared model without centralizing their raw data, offering an inherent level of privacy.<n> gradients and model updates can still leak sensitive information, while malicious servers may mount adversarial attacks such as Byzantine manipulation.<n>These vulnerabilities highlight the need to address differential privacy (DP) and Byzantine robustness within a unified framework.<n>We propose Byz-Clip21-SGD2M, a new algorithm that integrates robust aggregation with double momentum and carefully designed clipping.
  arXiv  Detail & Related papers  (2026-03-24T17:39:09Z)
• FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA [25.49850401602623]
  Federated LoRA provides a communication-efficient mechanism for fine-tuning large language models on decentralized data.<n>In practice, a discrepancy between the factor-wise averaging used to preserve low rank and the mathematically correct aggregation of local updates can cause significant aggregation error and unstable training.<n>We propose FedRot-LoRA, a framework that aligns client updates via transformations prior to aggregation.
  arXiv  Detail & Related papers  (2026-02-27T03:18:32Z)
• On the Geometric Coherence of Global Aggregation in Federated GNN [0.0]
  Federated Learning (FL) enables distributed training across multiple clients without centralized data sharing.<n> Graph Neural Networks (GNNs) model relational data through message passing.<n>In federated GNN settings, client graphs often exhibit heterogeneous structural and propagation characteristics.<n>Our work identifies a geometric failure mode of global aggregation in Cross- Domain Federated GNNs.<n>We propose GGRS, a server-side framework that regulates client updates prior to aggregation based on geometric admissibility criteria.
  arXiv  Detail & Related papers  (2026-02-17T11:34:04Z)
• Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models [63.70401095689976]
  We argue that replacing parameters with preferences represents a more scalable and privacy-preserving future.<n>We propose MoR, a federated alignment framework based on GRPO with Mixture-of-Rewards for heterogeneous VLMs.<n>MoR consistently outperforms federated alignment baselines in generalization, robustness, and cross-client adaptability.
  arXiv  Detail & Related papers  (2026-01-31T03:11:51Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation [60.81109086640437]
  We propose a novel framework called Federated Retrieval-Augmented Generation (FedE4RAG)<n>FedE4RAG facilitates collaborative training of client-side RAG retrieval models.<n>We apply homomorphic encryption within federated learning to safeguard model parameters.
  arXiv  Detail & Related papers  (2025-04-27T04:26:02Z)
• FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients [50.13097183691517]
  In real-world federated scenarios, there often exist a multitude of heterogeneous clients with varying computation and communication resources. We propose a novel federated tuning algorithm, FedRA. In each communication round, FedRA randomly generates an allocation matrix. It reorganizes a small number of layers from the original model based on the allocation matrix and fine-tunes using adapters.
  arXiv  Detail & Related papers  (2023-11-19T04:43:16Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.