Subspace Optimization for Efficient Federated Learning under Heterogeneous Data

• URL: http://arxiv.org/abs/2604.25467v1
• Date: Tue, 28 Apr 2026 10:12:57 GMT
• Title: Subspace Optimization for Efficient Federated Learning under Heterogeneous Data
• Abstract summary: Federated learning operates in a large-model regime where communication, memory, and computation are all scarce.<n>This paper proposes a subspace optimization method for federated learning (SSF)<n>SSF attains a non-asymptotic rate of order $widetildemathcalO (1/T+1/sqrtNKT)$.
• Score: 0.6999740786886536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning increasingly operates in a large-model regime where communication, memory, and computation are all scarce. Typically, non-IID client data induce drift that degrades the stability and performance of local training. Existing remedies such as SCAFFOLD introduce heterogeneity-correction mechanisms to address this challenge, but they incur substantial extra communication and memory overhead. This paper proposes a subspace optimization method for federated learning (SSF), which performs heterogeneity-corrected optimization in a low-dimensional subspace using only projected quantities, while preserving full-dimensional control information through a backfill-style update that retains residual components whenever the active subspace changes. Under standard smoothness and bounded-variance assumptions, SSF attains a non-asymptotic rate of order $\widetilde{\mathcal{O}}(1/T+1/\sqrt{NKT})$. Experiments show favorable accuracy--efficiency trade-offs under heterogeneous data.

Related papers

• Entropy-Regularized Probabilistic Gates for Sparse Model Discovery in Scarce-Data Federated Learning [0.0]
  Federated learning (FL) is a distributed machine learning (ML) paradigm with collaboration among multiple clients without sharing data.<n>We study entropy regularization of gate distributions as a mechanism to maintain uncertainty in sparse federated optimization.
  arXiv  Detail & Related papers  (2026-06-30T23:51:44Z)
• iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models [14.623213614166259]
  iGSP is a novel framework that achieves efficient adaptation via implicit gradient subspace projection.<n>Experiments on the MTIL benchmark demonstrate that iGSP achieves state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2026-05-19T03:22:14Z)
• Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration [56.074760766965085]
  PRISM achieves a dynamics-aware framework that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge.<n>Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.
  arXiv  Detail & Related papers  (2026-01-12T05:43:20Z)
• Regularizing Subspace Redundancy of Low-Rank Adaptation [54.473090597164834]
  We propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation.<n>Our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks.<n>As a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs.
  arXiv  Detail & Related papers  (2025-07-28T11:52:56Z)
• Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout [62.73150122809138]
  Federated Learning (FL) is a promising distributed machine learning approach that enables collaborative training of a global model using multiple edge devices.<n>We propose the FedDHAD FL framework, which comes with two novel methods: Dynamic Heterogeneous model aggregation (FedDH) and Adaptive Dropout (FedAD)<n>The combination of these two methods makes FedDHAD significantly outperform state-of-the-art solutions in terms of accuracy (up to 6.7% higher), efficiency (up to 2.02 times faster), and cost (up to 15.0% smaller)
  arXiv  Detail & Related papers  (2025-07-14T16:19:00Z)
• Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models [14.321060805197874]
  Large Language Models (LLMs) deployed in real-world settings increasingly face the need to unlearn sensitive, outdated, or proprietary information.<n>Existing unlearning methods formulate forgetting and retention as a regularized trade-off, combining both objectives into a single scalarized loss.<n>We propose a new formulation of LLM unlearning as a constrained optimization problem: forgetting is enforced via a novel logit-margin flattening loss.
  arXiv  Detail & Related papers  (2025-06-05T17:55:23Z)
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture.<n>Non-smooth regularization is often incorporated into machine learning tasks.<n>We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Handling Spatial-Temporal Data Heterogeneity for Federated Continual Learning via Tail Anchor [24.689188066180463]
  Federated continual learning (FCL) allows each client to continually update its knowledge from task streams.<n>We propose Federated Tail Anchor (FedTA) to mix trainable Tail Anchor with the frozen output features to adjust their position in the feature space.
  arXiv  Detail & Related papers  (2024-12-24T11:35:40Z)
• Learnable Sparse Customization in Heterogeneous Edge Computing [27.201987866208484]
  We propose Learnable Personalized Sparsification for heterogeneous Federated learning (FedLPS)<n>FedLPS learns the importance of model units on local data representation and derives an importance-based sparse pattern with minimals to accurately extract personalized data features.<n>Experiments show that FedLPS outperforms status quo approaches in accuracy and training costs.
  arXiv  Detail & Related papers  (2024-12-10T06:14:31Z)
• Analysis and Optimization of Wireless Federated Learning with Data Heterogeneity [72.85248553787538]
  This paper focuses on performance analysis and optimization for wireless FL, considering data heterogeneity, combined with wireless resource allocation. We formulate the loss function minimization problem, under constraints on long-term energy consumption and latency, and jointly optimize client scheduling, resource allocation, and the number of local training epochs (CRE) Experiments on real-world datasets demonstrate that the proposed algorithm outperforms other benchmarks in terms of the learning accuracy and energy consumption.
  arXiv  Detail & Related papers  (2023-08-04T04:18:01Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.