Differentially Private Deep Learning with Smooth Sensitivity

• URL: http://arxiv.org/abs/2003.00505v1
• Date: Sun, 1 Mar 2020 15:38:00 GMT
• Title: Differentially Private Deep Learning with Smooth Sensitivity
• Authors: Lichao Sun, Yingbo Zhou, Philip S. Yu, Caiming Xiong
• Abstract summary: We study privacy concerns through the lens of differential privacy. In this framework, privacy guarantees are generally obtained by perturbing models in such a way that specifics of data used to train the model are made ambiguous. One of the most important techniques used in previous works involves an ensemble of teacher models, which return information to a student based on a noisy voting procedure. In this work, we propose a novel voting mechanism with smooth sensitivity, which we call Immutable Noisy ArgMax, that, under certain conditions, can bear very large random noising from the teacher without affecting the useful information transferred to the student
• Score: 144.31324628007403
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ensuring the privacy of sensitive data used to train modern machine learning models is of paramount importance in many areas of practice. One approach to study these concerns is through the lens of differential privacy. In this framework, privacy guarantees are generally obtained by perturbing models in such a way that specifics of data used to train the model are made ambiguous. A particular instance of this approach is through a "teacher-student" framework, wherein the teacher, who owns the sensitive data, provides the student with useful, but noisy, information, hopefully allowing the student model to perform well on a given task without access to particular features of the sensitive data. Because stronger privacy guarantees generally involve more significant perturbation on the part of the teacher, deploying existing frameworks fundamentally involves a trade-off between student's performance and privacy guarantee. One of the most important techniques used in previous works involves an ensemble of teacher models, which return information to a student based on a noisy voting procedure. In this work, we propose a novel voting mechanism with smooth sensitivity, which we call Immutable Noisy ArgMax, that, under certain conditions, can bear very large random noising from the teacher without affecting the useful information transferred to the student. Compared with previous work, our approach improves over the state-of-the-art methods on all measures, and scale to larger tasks with both better performance and stronger privacy ($\epsilon \approx 0$). This new proposed framework can be applied with any machine learning models, and provides an appealing solution for tasks that requires training on a large amount of data.

Related papers

• Privacy-Preserving Student Learning with Differentially Private Data-Free Distillation [35.37005050907983]
  We present an effective teacher-student learning approach to train privacy-preserving deep learning models. Massive synthetic data can be generated for model training without exposing data privacy. A student is trained on the synthetic data with the supervision of private labels.
  arXiv  Detail & Related papers  (2024-09-19T01:00:18Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving [64.57963116462757]
  State-of-the-art methods usually follow the Teacher-Student' paradigm. Student model only has access to raw sensor data and conducts behavior cloning on the data collected by the teacher model. We propose DriveAdapter, which employs adapters with the feature alignment objective function between the student (perception) and teacher (planning) modules.
  arXiv  Detail & Related papers  (2023-08-01T09:21:53Z)
• A Survey on Differential Privacy with Machine Learning and Future Outlook [0.0]
  differential privacy is used to protect machine learning models from any attacks and vulnerabilities. This survey paper presents different differentially private machine learning algorithms categorized into two main categories.
  arXiv  Detail & Related papers  (2022-11-19T14:20:53Z)
• Private Semi-supervised Knowledge Transfer for Deep Learning from Noisy Labels [21.374707481630697]
  We propose PATE++, which combines the current advanced noisy label training mechanisms with the original PATE framework to enhance its accuracy. A novel structure of Generative Adversarial Nets (GANs) is developed in order to integrate them effectively. In addition, we develop a novel noisy label detection mechanism for semi-supervised model training to further improve student model performance when training with noisy labels.
  arXiv  Detail & Related papers  (2022-11-03T07:33:49Z)
• SF-PATE: Scalable, Fair, and Private Aggregation of Teacher Ensembles [50.90773979394264]
  This paper studies a model that protects the privacy of individuals' sensitive information while also allowing it to learn non-discriminatory predictors. A key characteristic of the proposed model is to enable the adoption of off-the-selves and non-private fair models to create a privacy-preserving and fair model.
  arXiv  Detail & Related papers  (2022-04-11T14:42:54Z)
• FaceLeaks: Inference Attacks against Transfer Learning Models via Black-box Queries [2.7564955518050693]
  We investigate if one can leak or infer private information without interacting with the teacher model directly. We propose novel strategies to infer from aggregate-level information. Our study indicates that information leakage is a real privacy threat to the transfer learning framework widely used in real-life situations.
  arXiv  Detail & Related papers  (2020-10-27T03:02:40Z)
• Differentially Private and Fair Deep Learning: A Lagrangian Dual Approach [54.32266555843765]
  This paper studies a model that protects the privacy of the individuals sensitive information while also allowing it to learn non-discriminatory predictors. The method relies on the notion of differential privacy and the use of Lagrangian duality to design neural networks that can accommodate fairness constraints.
  arXiv  Detail & Related papers  (2020-09-26T10:50:33Z)
• SPEED: Secure, PrivatE, and Efficient Deep learning [2.283665431721732]
  We introduce a deep learning framework able to deal with strong privacy constraints. Based on collaborative learning, differential privacy and homomorphic encryption, the proposed approach advances state-of-the-art.
  arXiv  Detail & Related papers  (2020-06-16T19:31:52Z)
• TIPRDC: Task-Independent Privacy-Respecting Data Crowdsourcing Framework for Deep Learning with Anonymized Intermediate Representations [49.20701800683092]
  We present TIPRDC, a task-independent privacy-respecting data crowdsourcing framework with anonymized intermediate representation. The goal of this framework is to learn a feature extractor that can hide the privacy information from the intermediate representations; while maximally retaining the original information embedded in the raw data for the data collector to accomplish unknown learning tasks.
  arXiv  Detail & Related papers  (2020-05-23T06:21:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.