SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training

• URL: http://arxiv.org/abs/2510.01447v2
• Date: Mon, 06 Oct 2025 20:27:34 GMT
• Title: SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training
• Authors: Dorsa Soleymani, Ali Dadsetan, Frank Rudzicz,
• Abstract summary: We introduce SoftAdaClip, a differentially private training method that replaces hard clipping with a smooth, tanh-based transformation.<n>We evaluate SoftAdaClip on various datasets, including MIMIC-III (clinical text), GOSSIS-eICU (structured healthcare), and Adult Income.<n>Our results show that SoftAdaClip reduces subgroup disparities by up to 87% compared to DP-SGD and up to 48% compared to Adaptive-DPSGD.
• Score: 13.525340904948829
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Differential privacy (DP) provides strong protection for sensitive data, but often reduces model performance and fairness, especially for underrepresented groups. One major reason is gradient clipping in DP-SGD, which can disproportionately suppress learning signals for minority subpopulations. Although adaptive clipping can enhance utility, it still relies on uniform hard clipping, which may restrict fairness. To address this, we introduce SoftAdaClip, a differentially private training method that replaces hard clipping with a smooth, tanh-based transformation to preserve relative gradient magnitudes while bounding sensitivity. We evaluate SoftAdaClip on various datasets, including MIMIC-III (clinical text), GOSSIS-eICU (structured healthcare), and Adult Income (tabular data). Our results show that SoftAdaClip reduces subgroup disparities by up to 87% compared to DP-SGD and up to 48% compared to Adaptive-DPSGD, and these reductions in subgroup disparities are statistically significant. These findings underscore the importance of integrating smooth transformations with adaptive mechanisms to achieve fair and private model training.

Related papers

• An Adaptive Differentially Private Federated Learning Framework with Bi-level Optimization [10.218291445871435]
  Federated learning enables collaborative model training across distributed clients while preserving data privacy.<n>In practical deployments, device heterogeneity, non-independent, and identically distributed (Non-IID) data often lead to highly unstable and biased gradient updates.<n>We propose an adaptive differentially private federated learning framework that explicitly targets model efficiency under heterogeneous and privacy-constrained settings.
  arXiv  Detail & Related papers  (2026-02-06T16:27:33Z)
• Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clipping [4.817614848684669]
  Differential privacy (DP) has become an essential framework for privacy-preserving machine learning.<n> Gradient clipping, which is often used in DP learning, can suppress larger gradients from challenging samples.<n>We show that this problem is amplified by adaptive clipping, which will often shrink the clipping bound to tiny values to match a well-fitting majority.<n>We propose bounded adaptive clipping, which introduces a tunable lower bound to prevent excessive gradient suppression.
  arXiv  Detail & Related papers  (2025-06-02T07:44:17Z)
• Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight [15.139854970044075]
  We introduce Differentially Private Per-sample Adaptive Scaling Clipping (DP-PSASC) This approach replaces traditional clipping with non-monotonous adaptive gradient scaling. Our theoretical and empirical analyses confirm that DP-PSASC preserves gradient privacy and delivers superior performance across diverse datasets.
  arXiv  Detail & Related papers  (2024-11-05T12:47:30Z)
• Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach [62.000948039914135]
  Using Differentially Private Gradient Descent with Gradient Clipping (DPSGD-GC) to ensure Differential Privacy (DP) comes at the cost of model performance degradation. We propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC. We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP.
  arXiv  Detail & Related papers  (2023-11-24T17:56:44Z)
• Sparsity-Preserving Differentially Private Training of Large Embedding Models [67.29926605156788]
  DP-SGD is a training algorithm that combines differential privacy with gradient descent. Applying DP-SGD naively to embedding models can destroy gradient sparsity, leading to reduced training efficiency. We present two new algorithms, DP-FEST and DP-AdaFEST, that preserve gradient sparsity during private training of large embedding models.
  arXiv  Detail & Related papers  (2023-11-14T17:59:51Z)
• Bias-Aware Minimisation: Understanding and Mitigating Estimator Bias in Private SGD [56.01810892677744]
  We show a connection between per-sample gradient norms and the estimation bias of the private gradient oracle used in DP-SGD. We propose Bias-Aware Minimisation (BAM) that allows for the provable reduction of private gradient estimator bias.
  arXiv  Detail & Related papers  (2023-08-23T09:20:41Z)
• Disparate Impact in Differential Privacy from Gradient Misalignment [0.8192907805418583]
  We study the fine-grained causes of unfairness in DPSGD and identify gradient misalignment due to inequitable gradient clipping. This observation leads us to a new method for reducing unfairness by preventing gradient misalignment in DPSGD.
  arXiv  Detail & Related papers  (2022-06-15T18:06:45Z)
• Large Scale Transfer Learning for Differentially Private Image Classification [51.10365553035979]
  Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy. Private training using DP-SGD protects against leakage by injecting noise into individual example gradients. While this result is quite appealing, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training.
  arXiv  Detail & Related papers  (2022-05-06T01:22:20Z)
• Understanding Gradient Clipping in Private SGD: A Geometric Perspective [68.61254575987013]
  Deep learning models are increasingly popular in many machine learning applications where the training data may contain sensitive information. Many learning systems now incorporate differential privacy by training their models with (differentially) private SGD. A key step in each private SGD update is gradient clipping that shrinks the gradient of an individual example whenever its L2 norm exceeds some threshold.
  arXiv  Detail & Related papers  (2020-06-27T19:08:12Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)
• Removing Disparate Impact of Differentially Private Stochastic Gradient Descent on Model Accuracy [18.69118059633505]
  When we enforce differential privacy in machine learning, the utility-privacy trade-off is different w.r.t. each group. In this work, we analyze the inequality in utility loss by differential privacy and propose a modified differentially private gradient descent (DPSGD) Our experimental evaluation shows how group sample size and group clipping bias affect the impact of differential privacy in DPSGD, and how adaptive clipping for each group helps to mitigate the disparate impact caused by differential privacy in DPSGDF.
  arXiv  Detail & Related papers  (2020-03-08T02:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.