DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Data

• URL: http://arxiv.org/abs/2606.03209v1
• Date: Tue, 02 Jun 2026 06:08:31 GMT
• Title: DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Data
• Abstract summary: Decentralized fine-tuning for large language models is challenging in privacy-sensitive environments.<n>We propose DECA, a resource-efficient decentralized FPFT framework for LLMs on non-IID data.<n>We show that DECA achieves fast convergence, strong downstream performance, and significant resource efficiency.
• Score: 20.24370872176655
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-tuning large language models (LLMs) in privacy-sensitive and resource-constrained environments remains challenging. Since training data are often distributed across multiple clients, decentralized fine-tuning offers a natural paradigm for collaborative adaptation without a central server. However, enabling full-parameter fine-tuning (FPFT) in this decentralized setting is difficult: FPFT provides strong adaptation capacity but incurs prohibitive resource consumption for billion-scale models. Existing decentralized LLM fine-tuning methods therefore mainly rely on parameter-efficient updates, which improve efficiency but may restrict downstream performance. Moreover, client data are typically non-IID, making decentralized optimization more vulnerable to client drift and unstable convergence. To address these challenges, we propose DECA, a resource-efficient decentralized FPFT framework for LLMs on non-IID data. DECA partitions model parameters into disjoint blocks and performs sequential block-wise Adam optimization, reducing resource consumption while preserving decentralized full-parameter adaptation. To stabilize training, DECA further introduces first- and second-order block-wise moment estimates with fresh local gradient statistics and consensus-derived discrepancy signals. We provide rigorous theoretical analysis and extensive experiments, showing that DECA achieves fast convergence, strong downstream performance, and significant resource efficiency.

Related papers

• DUET: Decentralized Bilevel Optimization without Lower-Level Strong Convexity [77.51981761667302]
  We introduce a new single-loop DBO algorithm called diminishing quadratically-regularized bilevel decentralized optimization (DUET)<n>DUET eliminates the need for LLSC by introducing a diminishing quadratic regularization to the lower-level (LL) objective.<n>To the best of our knowledge, this is the first work to tackle DBO without LLSC under decentralized settings with data heterogeneity.
  arXiv  Detail & Related papers  (2026-06-19T06:45:13Z)
• AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environments [7.335282879368365]
  Large Language Models (LLMs) have been widely deployed across various scenarios, including autonomous driving, industrial inspection, and personalized IoT services.<n>This paper proposes AlignFed, an asynchronous federated fine-tuning framework for LLMs tailored to heterogeneous edge environments.
  arXiv  Detail & Related papers  (2026-06-06T14:28:36Z)
• FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment [55.97027207627]
  Fed PDPO (Federated Personalized Direct Preference Optimization) is a personalized framework for preference alignment of large language models (LLMs)<n>It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation.<n>Experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.
  arXiv  Detail & Related papers  (2026-03-20T08:24:49Z)
• FedZMG: Efficient Client-Side Optimization in Federated Learning [0.19116784879310023]
  Federated Zero Mean Gradients (FedZMG) is a parameter-free, client-side optimization algorithm designed to tackle client-drift.<n>FedZMG projects local gradients onto a zero-mean hyperplane, effectively neutralizing the "intensity" or "bias" shifts inherent in heterogeneous data distributions.
  arXiv  Detail & Related papers  (2026-02-20T17:45:28Z)
• Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detection [53.45696787935487]
  Federated Learning (FL) enables collaborative model training across large-scale distributed service nodes.<n>In real-world service-oriented deployments, data generated by heterogeneous users, devices, and application scenarios are inherently non-IID.<n>We propose FLood, a novel FL framework inspired by out-of-distribution (OOD) detection.
  arXiv  Detail & Related papers  (2026-02-01T05:54:59Z)
• AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption [3.805501490912696]
  Federated fine-tuning has emerged as a promising approach to adapt foundation models to downstream tasks using decentralized data.<n>We propose AFLoRA, an adaptive and lightweight federated fine-tuning framework for Large Language Models.
  arXiv  Detail & Related papers  (2025-05-30T16:35:32Z)
• Decentralized Low-Rank Fine-Tuning of Large Language Models [12.270878920401948]
  We propose Dec-LoRA, a decentralized fine-tuning algorithm for Large Language Models (LLMs) based Low-Rank Adaptation (LoRA)<n>Through experiments on BERT and LLaMA, we demonstrate that Dec-LoRA achieves comparable performance to centralized LoRA under various conditions.<n>These findings highlight the potential of Dec-LoRA for scalable fine-tuning in decentralized environments.
  arXiv  Detail & Related papers  (2025-01-26T01:56:25Z)
• Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization [71.87335804334616]
  Federated learning (FL) is a promising paradigm to enable collaborative model training with decentralized data. The training process of Large Language Models (LLMs) generally incurs the update of significant parameters. This paper proposes an efficient partial prompt tuning approach to improve performance and efficiency simultaneously.
  arXiv  Detail & Related papers  (2023-10-23T16:37:59Z)
• Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training [67.67045085186797]
  Almost all existing systems have to face large communication burdens if the central FL server fails. It personalizes the "right" in the deep models by alternately updating the shared and personal parameters. To further promote the shared parameters aggregation process, we propose DFed integrating the local Sharpness Miniization.
  arXiv  Detail & Related papers  (2023-05-24T13:52:18Z)
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels [69.18343801164741]
  partitioned edge learning (PARTEL) implements parameter-server training, a well known distributed learning method, in wireless network. We consider the case of deep neural network (DNN) models which can be trained using PARTEL by introducing some auxiliary variables.
  arXiv  Detail & Related papers  (2020-10-08T15:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.