Hidden State Differential Private Mini-Batch Block Coordinate Descent for Multi-convexity Optimization

• URL: http://arxiv.org/abs/2407.08233v2
• Date: Sat, 31 May 2025 18:08:15 GMT
• Title: Hidden State Differential Private Mini-Batch Block Coordinate Descent for Multi-convexity Optimization
• Authors: Ding Chen, Chen Liu,
• Abstract summary: We investigate the differential privacy guarantees under the hidden state assumption (HSA) for multi- bound problems.<n>Recent analyses of privacy loss under hidden state assumption have relied on strong assumptions such as convexity.<n>In addition to a gradient for privacy loss, theoretical analysis is also compatible with neural descent and adaptive noise scenarios.
• Score: 2.3371504588528635
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate the differential privacy (DP) guarantees under the hidden state assumption (HSA) for multi-convex problems. Recent analyses of privacy loss under the hidden state assumption have relied on strong assumptions such as convexity, thereby limiting their applicability to practical problems. In this paper, we introduce the Differential Privacy Mini-Batch Block Coordinate Descent (DP-MBCD) algorithm, accompanied by the privacy loss accounting methods under the hidden state assumption. Our proposed methods apply to a broad range of classical non-convex problems which are or can be converted to multi-convex problems, such as matrix factorization and neural network training. In addition to a tighter bound for privacy loss, our theoretical analysis is also compatible with proximal gradient descent and adaptive calibrated noise scenarios.

Related papers

• Private Rate-Constrained Optimization with Applications to Fair Learning [39.172158806012966]
  We study constrained problems under differential privacy (DP)<n>We develop RaCO-DP, a variant of the Gradient Descent-Ascent (SGDA) algorithm which solves the Lagrangian formulation of rate constraint problems.
  arXiv  Detail & Related papers  (2025-05-28T17:55:01Z)
• Forward Learning with Differential Privacy [27.164507868291913]
  We propose a blueprivatized forward-learning algorithm, Differential Private Unified Likelihood Ratio (DP-ULR) Our experiments indicate that DP-ULR achieves competitive performance compared to traditional differential privacy training algorithms based on backpropagation.
  arXiv  Detail & Related papers  (2025-04-01T04:14:53Z)
• Linear-Time User-Level DP-SCO via Robust Statistics [55.350093142673316]
  User-level differentially private convex optimization (DP-SCO) has garnered significant attention due to the importance of safeguarding user privacy in machine learning applications.<n>Current methods, such as those based on differentially private gradient descent (DP-SGD), often struggle with high noise accumulation and suboptimal utility.<n>We introduce a novel linear-time algorithm that leverages robust statistics, specifically the median and trimmed mean, to overcome these challenges.
  arXiv  Detail & Related papers  (2025-02-13T02:05:45Z)
• A Neural Network Training Method Based on Neuron Connection Coefficient Adjustments [2.684545081600664]
  We present a new training approach for a neural network based on symmetric differential equations. Unlike the previously introduced method, this approach does not require adjustments to the fixed points of the differential equations. To validate this approach, we tested it on the MNIST dataset and achieved promising results.
  arXiv  Detail & Related papers  (2025-01-25T16:07:30Z)
• Minimax Optimal Two-Sample Testing under Local Differential Privacy [3.3317825075368908]
  We explore the trade-off between privacy and statistical utility in private two-sample testing under local differential privacy (LDP) We introduce private permutation tests using practical privacy mechanisms such as Laplace, discrete Laplace, and Google's RAPPOR. We study continuous data via binning and study its uniform separation rates under LDP over H"older and Besov smoothness classes.
  arXiv  Detail & Related papers  (2024-11-13T22:44:25Z)
• Private Language Models via Truncated Laplacian Mechanism [18.77713904999236]
  We propose a novel private embedding method called the high dimensional truncated Laplacian mechanism. We show that our method has a lower variance compared to the previous private word embedding methods. Remarkably, even in the high privacy regime, our approach only incurs a slight decrease in utility compared to the non-private scenario.
  arXiv  Detail & Related papers  (2024-10-10T15:25:02Z)
• Approximating Two-Layer ReLU Networks for Hidden State Analysis in Differential Privacy [3.8254443661593633]
  We show that it is possible to privately train convex problems with privacy-utility trade-offs comparable to those of one hidden-layer ReLU networks trained with DP-SGD. Our experiments on benchmark classification tasks show that NoisyCGD can achieve privacy-utility trade-offs comparable to DP-SGD applied to one-hidden-layer ReLU networks.
  arXiv  Detail & Related papers  (2024-07-05T22:43:32Z)
• Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks [72.51255282371805]
  We prove a privacy bound for the KL divergence between model distributions on worst-case neighboring datasets. We find that this KL privacy bound is largely determined by the expected squared gradient norm relative to model parameters during training.
  arXiv  Detail & Related papers  (2023-10-31T16:13:22Z)
• DP-SGD with weight clipping [1.0878040851638]
  We present a novel approach that mitigates the bias arising from traditional gradient clipping. By leveraging a public upper bound of the Lipschitz value of the current model and its current location within the search domain, we can achieve refined noise level adjustments.
  arXiv  Detail & Related papers  (2023-10-27T09:17:15Z)
• Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters [69.24377241408851]
  Overfitting to the source domain is a common issue in gradient-based training of deep neural networks. We propose to base the selection on gradient-signal-to-noise ratio (GSNR) of network's parameters.
  arXiv  Detail & Related papers  (2023-10-11T10:21:34Z)
• On Differential Privacy and Adaptive Data Analysis with Bounded Space [76.10334958368618]
  We study the space complexity of the two related fields of differential privacy and adaptive data analysis. We show that there exists a problem P that requires exponentially more space to be solved efficiently with differential privacy. The line of work on adaptive data analysis focuses on understanding the number of samples needed for answering a sequence of adaptive queries.
  arXiv  Detail & Related papers  (2023-02-11T14:45:31Z)
• Local Graph-homomorphic Processing for Privatized Distributed Systems [57.14673504239551]
  We show that the added noise does not affect the performance of the learned model. This is a significant improvement to previous works on differential privacy for distributed algorithms.
  arXiv  Detail & Related papers  (2022-10-26T10:00:14Z)
• Differentially Private Stochastic Gradient Descent with Low-Noise [49.981789906200035]
  Modern machine learning algorithms aim to extract fine-grained information from data to provide accurate predictions, which often conflicts with the goal of privacy protection. This paper addresses the practical and theoretical importance of developing privacy-preserving machine learning algorithms that ensure good performance while preserving privacy.
  arXiv  Detail & Related papers  (2022-09-09T08:54:13Z)
• Differentially Private Learning Needs Hidden State (Or Much Faster Convergence) [9.429448411561541]
  We show that differentially private learning, with a tight bound, needs hidden state privacy analysis or a fast convergence. Our converging privacy analysis, thus, shows that differentially private learning, with a tight bound, needs hidden state privacy analysis or a fast convergence.
  arXiv  Detail & Related papers  (2022-03-10T13:31:08Z)
• Differentially Private SGDA for Minimax Problems [83.57322009102973]
  We prove that gradient descent ascent (SGDA) can achieve optimal utility in terms of weak primal-dual population risk. This is the first-ever-known result for non-smoothly-strongly-concave setting.
  arXiv  Detail & Related papers  (2022-01-22T13:05:39Z)
• NeuralDP Differentially private neural networks by design [61.675604648670095]
  We propose NeuralDP, a technique for privatising activations of some layer within a neural network. We experimentally demonstrate on two datasets that our method offers substantially improved privacy-utility trade-offs compared to DP-SGD.
  arXiv  Detail & Related papers  (2021-07-30T12:40:19Z)
• Differentially private training of neural networks with Langevin dynamics forcalibrated predictive uncertainty [58.730520380312676]
  We show that differentially private gradient descent (DP-SGD) can yield poorly calibrated, overconfident deep learning models. This represents a serious issue for safety-critical applications, e.g. in medical diagnosis.
  arXiv  Detail & Related papers  (2021-07-09T08:14:45Z)
• Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent [95.94432031144716]
  We propose a unified non- optimization framework for the analysis of a learning network. We show that existing guarantees can be trained unified through gradient descent.
  arXiv  Detail & Related papers  (2021-06-25T17:45:00Z)
• Wide Network Learning with Differential Privacy [7.453881927237143]
  Current generation of neural networks suffers significant loss accuracy under most practically relevant privacy training regimes. We develop a general approach towards training these models that takes advantage of the sparsity of the gradients of private Empirical Minimization (ERM) Following the same number of parameters, we propose a novel algorithm for privately training neural networks.
  arXiv  Detail & Related papers  (2021-03-01T20:31:50Z)
• Robustness Threats of Differential Privacy [70.818129585404]
  We experimentally demonstrate that networks, trained with differential privacy, in some settings might be even more vulnerable in comparison to non-private versions. We study how the main ingredients of differentially private neural networks training, such as gradient clipping and noise addition, affect the robustness of the model.
  arXiv  Detail & Related papers  (2020-12-14T18:59:24Z)
• Differentially Private Deep Learning with Direct Feedback Alignment [15.410557873153833]
  We propose the first differentially private method for training deep neural networks with direct feedback alignment (DFA) DFA achieves significant gains in accuracy (often by 10-20%) compared to backprop-based differentially private training on a variety of architectures.
  arXiv  Detail & Related papers  (2020-10-08T00:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.