Differentially Private Iterative Screening Rules for Linear Regression

• URL: http://arxiv.org/abs/2502.18578v1
• Date: Tue, 25 Feb 2025 19:06:19 GMT
• Title: Differentially Private Iterative Screening Rules for Linear Regression
• Authors: Amol Khanna, Fred Lu, Edward Raff,
• Abstract summary: In this paper, we develop the first private screening rule for linear regression.<n>We find that this screening rule is too strong: it screens too many coefficients as a result of the private screening step.<n>However, a weakened implementation of private screening reduces overscreening and improves performance.
• Score: 45.50668718813776
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Linear $L_1$-regularized models have remained one of the simplest and most effective tools in data science. Over the past decade, screening rules have risen in popularity as a way to eliminate features when producing the sparse regression weights of $L_1$ models. However, despite the increasing need of privacy-preserving models for data analysis, to the best of our knowledge, no differentially private screening rule exists. In this paper, we develop the first private screening rule for linear regression. We initially find that this screening rule is too strong: it screens too many coefficients as a result of the private screening step. However, a weakened implementation of private screening reduces overscreening and improves performance.

Related papers

• Near-Optimal Private Learning in Linear Contextual Bandits [61.39697409886124]
  We analyze the problem of private learning in generalized linear contextual bandits.<n>Our results imply that joint privacy is almost "for free" in all the settings we consider.
  arXiv  Detail & Related papers  (2025-02-18T18:35:24Z)
• Optimized Tradeoffs for Private Prediction with Majority Ensembling [59.99331405291337]
  We introduce the Data-dependent Randomized Response Majority (DaRRM) algorithm.<n>DaRRM is parameterized by a data-dependent noise function $gamma$, and enables efficient utility optimization over the class of all private algorithms.<n>We show that DaRRM provably enjoys a privacy gain of a factor of 2 over common baselines, with fixed utility.
  arXiv  Detail & Related papers  (2024-11-27T00:48:48Z)
• Privacy for Free in the Over-Parameterized Regime [19.261178173399784]
  Differentially private gradient descent (DP-GD) is a popular algorithm to train deep learning models with provable guarantees on the privacy of the training data. In this work, we show that in the popular random features model with quadratic loss, for any sufficiently large $p$, privacy can be obtained for free, i.e., $left|R_P right| = o(1)$, not only when the privacy parameter $varepsilon$ has constant order, but also in the strongly private setting $varepsilon = o(1)$.
  arXiv  Detail & Related papers  (2024-10-18T18:01:11Z)
• Privacy Profiles for Private Selection [21.162924003105484]
  We work out an easy-to-use recipe that bounds privacy profiles of ReportNoisyMax and PrivateTuning using the privacy profiles of the base algorithms they corral. Our approach improves over all regimes of interest and leads to substantial benefits in end-to-end private learning experiments.
  arXiv  Detail & Related papers  (2024-02-09T08:31:46Z)
• Private Fine-tuning of Large Language Models with Zeroth-order Optimization [51.19403058739522]
  Differentially private gradient descent (DP-SGD) allows models to be trained in a privacy-preserving manner.<n>We introduce DP-ZO, a private fine-tuning framework for large language models by privatizing zeroth order optimization methods.
  arXiv  Detail & Related papers  (2024-01-09T03:53:59Z)
• Differentially Private Statistical Inference through $\beta$-Divergence One Posterior Sampling [2.8544822698499255]
  We propose a posterior sampling scheme from a generalised posterior targeting the minimisation of the $beta$-divergence between the model and the data generating process. This provides private estimation that is generally applicable without requiring changes to the underlying model. We show that $beta$D-Bayes produces more precise inference estimation for the same privacy guarantees.
  arXiv  Detail & Related papers  (2023-07-11T12:00:15Z)
• The Challenge of Differentially Private Screening Rules [32.18582226044492]
  We develop the first differentially private screening rule for linear and logistic regression. In doing so, we discover difficulties in the task of making a useful private screening rule due to the amount of noise added to ensure privacy.
  arXiv  Detail & Related papers  (2023-03-18T01:45:34Z)
• Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent [69.14164921515949]
  We characterize privacy guarantees for individual examples when releasing models trained by DP-SGD. We find that most examples enjoy stronger privacy guarantees than the worst-case bound. This implies groups that are underserved in terms of model utility simultaneously experience weaker privacy guarantees.
  arXiv  Detail & Related papers  (2022-06-06T13:49:37Z)
• The Hessian Screening Rule [5.076419064097734]
  Hessian Screening Rule uses second-order information from the model to provide more accurate screening. We show that the rule outperforms all other alternatives in simulated experiments with high correlation.
  arXiv  Detail & Related papers  (2021-04-27T07:55:29Z)
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning [74.73901662374921]
  A differentially private model degrades the utility drastically when the model comprises a large number of trainable parameters. We propose an algorithm emphGradient Embedding Perturbation (GEP) towards training differentially private deep models with decent accuracy.
  arXiv  Detail & Related papers  (2021-02-25T04:29:58Z)
• Fast OSCAR and OWL Regression via Safe Screening Rules [97.28167655721766]
  Ordered $L_1$ (OWL) regularized regression is a new regression analysis for high-dimensional sparse learning. Proximal gradient methods are used as standard approaches to solve OWL regression. We propose the first safe screening rule for OWL regression by exploring the order of the primal solution with the unknown order structure.
  arXiv  Detail & Related papers  (2020-06-29T23:35:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.