Differential Privacy Mechanisms in Neural Tangent Kernel Regression

• URL: http://arxiv.org/abs/2407.13621v2
• Date: Sat, 2 Nov 2024 05:30:20 GMT
• Title: Differential Privacy Mechanisms in Neural Tangent Kernel Regression
• Authors: Jiuxiang Gu, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song,
• Abstract summary: We study differential privacy (DP) in the Neural Tangent Kernel (NTK) regression setting. We show provable guarantees for both differential privacy and test accuracy of our NTK regression. To our knowledge, this is the first work to provide a DP guarantee for NTK regression.
• Score: 29.187250620950927
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training data privacy is a fundamental problem in modern Artificial Intelligence (AI) applications, such as face recognition, recommendation systems, language generation, and many others, as it may contain sensitive user information related to legal issues. To fundamentally understand how privacy mechanisms work in AI applications, we study differential privacy (DP) in the Neural Tangent Kernel (NTK) regression setting, where DP is one of the most powerful tools for measuring privacy under statistical learning, and NTK is one of the most popular analysis frameworks for studying the learning mechanisms of deep neural networks. In our work, we can show provable guarantees for both differential privacy and test accuracy of our NTK regression. Furthermore, we conduct experiments on the basic image classification dataset CIFAR10 to demonstrate that NTK regression can preserve good accuracy under a modest privacy budget, supporting the validity of our analysis. To our knowledge, this is the first work to provide a DP guarantee for NTK regression.

Related papers

• Privacy-Preserving Hybrid Ensemble Model for Network Anomaly Detection: Balancing Security and Data Protection [6.5920909061458355]
  We propose a hybrid ensemble model that incorporates privacy-preserving techniques to address both detection accuracy and data protection. Our model combines the strengths of several machine learning algo- rithms, including K-Nearest Neighbors (KNN), Support Vector Machines (SVM), XGBoost, and Artificial Neural Networks (ANN)
  arXiv  Detail & Related papers  (2025-02-13T06:33:16Z)
• Differentially Private Random Feature Model [52.468511541184895]
  We produce a differentially private random feature model for privacy-preserving kernel machines. We show that our method preserves privacy and derive a generalization error bound for the method.
  arXiv  Detail & Related papers  (2024-12-06T05:31:08Z)
• Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks [72.51255282371805]
  We prove a privacy bound for the KL divergence between model distributions on worst-case neighboring datasets. We find that this KL privacy bound is largely determined by the expected squared gradient norm relative to model parameters during training.
  arXiv  Detail & Related papers  (2023-10-31T16:13:22Z)
• Discriminative Adversarial Privacy: Balancing Accuracy and Membership Privacy in Neural Networks [7.0895962209555465]
  Discriminative Adversarial Privacy (DAP) is a learning technique designed to achieve a balance between model performance, speed, and privacy. DAP relies on adversarial training based on a novel loss function able to minimise the prediction error while maximising the MIA's error. In addition, we introduce a novel metric named Accuracy Over Privacy (AOP) to capture the performance-privacy trade-off.
  arXiv  Detail & Related papers  (2023-06-05T17:25:45Z)
• Differentially Private Stochastic Gradient Descent with Low-Noise [49.981789906200035]
  Modern machine learning algorithms aim to extract fine-grained information from data to provide accurate predictions, which often conflicts with the goal of privacy protection. This paper addresses the practical and theoretical importance of developing privacy-preserving machine learning algorithms that ensure good performance while preserving privacy.
  arXiv  Detail & Related papers  (2022-09-09T08:54:13Z)
• Sensitivity analysis in differentially private machine learning using hybrid automatic differentiation [54.88777449903538]
  We introduce a novel textithybrid automatic differentiation (AD) system for sensitivity analysis. This enables modelling the sensitivity of arbitrary differentiable function compositions, such as the training of neural networks on private data. Our approach can enable the principled reasoning about privacy loss in the setting of data processing.
  arXiv  Detail & Related papers  (2021-07-09T07:19:23Z)
• DiPSeN: Differentially Private Self-normalizing Neural Networks For Adversarial Robustness in Federated Learning [6.1448102196124195]
  Federated learning has proven to help protect against privacy violations and information leakage. It introduces new risk vectors which make machine learning models more difficult to defend against adversarial samples. We introduce DiPSeN, a Differentially Private Self-normalizing Neural Network which combines elements of differential privacy noise with self-normalizing techniques.
  arXiv  Detail & Related papers  (2021-01-08T20:49:56Z)
• Robustness Threats of Differential Privacy [70.818129585404]
  We experimentally demonstrate that networks, trained with differential privacy, in some settings might be even more vulnerable in comparison to non-private versions. We study how the main ingredients of differentially private neural networks training, such as gradient clipping and noise addition, affect the robustness of the model.
  arXiv  Detail & Related papers  (2020-12-14T18:59:24Z)
• Tempered Sigmoid Activations for Deep Learning with Differential Privacy [33.574715000662316]
  We show that the choice of activation function is central to bounding the sensitivity of privacy-preserving deep learning. We achieve new state-of-the-art accuracy on MNIST, FashionMNIST, and CIFAR10 without any modification of the learning procedure fundamentals.
  arXiv  Detail & Related papers  (2020-07-28T13:19:45Z)
• CryptoSPN: Privacy-preserving Sum-Product Network Inference [84.88362774693914]
  We present a framework for privacy-preserving inference of sum-product networks (SPNs) CryptoSPN achieves highly efficient and accurate inference in the order of seconds for medium-sized SPNs.
  arXiv  Detail & Related papers  (2020-02-03T14:49:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.