DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models

• URL: http://arxiv.org/abs/2605.10272v1
• Date: Mon, 11 May 2026 09:32:21 GMT
• Title: DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
• Abstract summary: Federated learning (FL) enables the collaborative training of large-scale language models across edge devices while keeping user data on-device.<n>DP-SGD mitigates this risk by clipping each client's contribution to a threshold $C$ and adding noise proportional to $C$.<n>We introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation.<n> Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of $6.6%$
• Score: 25.94292199187493
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provided gradients. Differentially private stochastic gradient descent (DP-SGD) mitigates this risk by clipping each client's contribution to a threshold $C$ and adding noise proportional to $C$. Existing adaptive clipping techniques dynamically adjust $C$ but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of $6.6\%$.

Related papers

• DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum [13.705460306140004]
  Differentially private gradient descent (DP-SGD) has become the standard framework for privacy-preserving machine learning.<n>We propose Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum (DP-MacAdam)
  arXiv  Detail & Related papers  (2026-06-03T20:58:34Z)
• PAC-DP: Personalized Adaptive Clipping for Differentially Private Federated Learning [16.9867659381048]
  PAC-DP is a personalized Adaptive Clipping framework for federated learning.<n>We show that PAC-DP surpasses conventional fixed-threshold approaches under matched privacy budgets.
  arXiv  Detail & Related papers  (2026-03-25T07:06:33Z)
• Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning [0.9861588522527782]
  We present RLDP, the first framework to cast DP optimization itself as a closed-loop control problem amenable to modern deep reinforcement learning (RL)<n>Across more than 1,600 experiments on GPT2-small, Llama-1B, Llama-3B, and Mistral-7B, RLDP delivers perplexity reductions of 1.3-3.0.5% and an average 5.6% downstream utility gain.
  arXiv  Detail & Related papers  (2025-07-30T10:46:53Z)
• Smoothed Normalization for Efficient Distributed Private Optimization [54.197255548244705]
  Federated learning enables machine learning models with privacy of participants.<n>There is no differentially private distributed method for training, non-feedback problems.<n>We introduce a new distributed algorithm $alpha$-$sf NormEC$ with provable convergence guarantees.
  arXiv  Detail & Related papers  (2025-02-19T07:10:32Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Differentially Private Learning with Per-Sample Adaptive Clipping [8.401653565794353]
  We propose a Differentially Private Per-Sample Adaptive Clipping (DP-PSAC) algorithm based on a non-monotonic adaptive weight function. We show that DP-PSAC outperforms or matches the state-of-the-art methods on multiple main-stream vision and language tasks.
  arXiv  Detail & Related papers  (2022-12-01T07:26:49Z)
• Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization [94.06564567766475]
  DP-SGD and DP-NSGD mitigate the risk of large models memorizing sensitive training data. We show that these two algorithms achieve similar best accuracy while DP-NSGD is comparatively easier to tune than DP-SGD.
  arXiv  Detail & Related papers  (2022-06-27T03:45:02Z)
• Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger [39.93710312222771]
  Per-example clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. We propose an easy-to-use replacement, called automatic clipping, that eliminates the need to tune R for any DPs.
  arXiv  Detail & Related papers  (2022-06-14T19:49:44Z)
• Large Scale Transfer Learning for Differentially Private Image Classification [51.10365553035979]
  Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy. Private training using DP-SGD protects against leakage by injecting noise into individual example gradients. While this result is quite appealing, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training.
  arXiv  Detail & Related papers  (2022-05-06T01:22:20Z)
• Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds [72.63031036770425]
  We propose differentially private (DP) algorithms for bound non-dimensional optimization. We demonstrate two popular deep learning methods on the empirical advantages over standard gradient methods.
  arXiv  Detail & Related papers  (2020-06-24T06:01:24Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.