Scalable Differentially Private Bayesian Optimization

• URL: http://arxiv.org/abs/2502.06044v2
• Date: Wed, 28 May 2025 20:06:18 GMT
• Title: Scalable Differentially Private Bayesian Optimization
• Authors: Getoar Sopa, Juraj Marusic, Marco Avella-Medina, John P. Cunningham,
• Abstract summary: We develop a method to privately optimize potentially high-dimensional parameter spaces.<n>Our theoretical results show that our method converges exponentially fast to a locally optimal parameter configuration.<n>We prove that our algorithm maintains privacy and empirically display superior performance to existing methods.
• Score: 17.28046301424826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, there has been much work on scaling Bayesian Optimization to high-dimensional problems, for example hyperparameter tuning in large machine learning models. These scalable methods have been successful, finding high objective values much more quickly than traditional global Bayesian Optimization or random search-based methods. At the same time, these large models often use sensitive data, but preservation of Differential Privacy has not scaled alongside these modern Bayesian Optimization procedures. Here we develop a method to privately optimize potentially high-dimensional parameter spaces using privatized Gradient Informative Bayesian Optimization. Our theoretical results show that under suitable conditions, our method converges exponentially fast to a locally optimal parameter configuration, up to a natural privacy error. Moreover, regardless of whether the assumptions are satisfied, we prove that our algorithm maintains privacy and empirically display superior performance to existing methods in the high-dimensional hyperparameter setting.

Related papers

• Scaling Exponents Across Parameterizations and Optimizers [94.54718325264218]
  We propose a new perspective on parameterization by investigating a key assumption in prior work. Our empirical investigation includes tens of thousands of models trained with all combinations of threes. We find that the best learning rate scaling prescription would often have been excluded by the assumptions in prior work.
  arXiv  Detail & Related papers  (2024-07-08T12:32:51Z)
• Differentially Private Optimization with Sparse Gradients [60.853074897282625]
  We study differentially private (DP) optimization problems under sparsity of individual gradients. Building on this, we obtain pure- and approximate-DP algorithms with almost optimal rates for convex optimization with sparse gradients.
  arXiv  Detail & Related papers  (2024-04-16T20:01:10Z)
• End-to-End Learning for Fair Multiobjective Optimization Under Uncertainty [55.04219793298687]
  The Predict-Then-Forecast (PtO) paradigm in machine learning aims to maximize downstream decision quality. This paper extends the PtO methodology to optimization problems with nondifferentiable Ordered Weighted Averaging (OWA) objectives. It shows how optimization of OWA functions can be effectively integrated with parametric prediction for fair and robust optimization under uncertainty.
  arXiv  Detail & Related papers  (2024-02-12T16:33:35Z)
• Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter $ε$ via Gradient Magnitude Histogram Analysis [0.7366405857677226]
  We introduce a new framework based on the empirical probability density function of the loss's magnitude, termed the "gradient magnitude histogram" We propose a novel algorithm using gradient magnitude histograms to automatically estimate a refined and accurate search space for the optimal safeguard.
  arXiv  Detail & Related papers  (2023-11-20T04:34:19Z)
• Online Sensitivity Optimization in Differentially Private Learning [8.12606646175019]
  We present a novel approach to dynamically optimize the clipping threshold. We treat this threshold as an additional learnable parameter, establishing a clean relationship between the threshold and the cost function. Our method is thoroughly assessed against alternative fixed and adaptive strategies across diverse datasets, tasks, model dimensions, and privacy levels.
  arXiv  Detail & Related papers  (2023-10-02T00:30:49Z)
• DP-HyPO: An Adaptive Private Hyperparameter Optimization Framework [31.628466186344582]
  We introduce DP-HyPO, a pioneering framework for adaptive'' private hyperparameter optimization. We provide a comprehensive differential privacy analysis of our framework. We empirically demonstrate the effectiveness of DP-HyPO on a diverse set of real-world datasets.
  arXiv  Detail & Related papers  (2023-06-09T07:55:46Z)
• Large-Batch, Iteration-Efficient Neural Bayesian Design Optimization [37.339567743948955]
  We present a novel Bayesian optimization framework specifically tailored to address the limitations of BO. Our key contribution is a highly scalable, sample-based acquisition function that performs a non-dominated sorting of objectives. We show that our acquisition function in combination with different Bayesian neural network surrogates is effective in data-intensive environments with a minimal number of iterations.
  arXiv  Detail & Related papers  (2023-06-01T19:10:57Z)
• A Survey on Multi-Objective based Parameter Optimization for Deep Learning [1.3223682837381137]
  We focus on exploring the effectiveness of multi-objective optimization strategies for parameter optimization in conjunction with deep neural networks. The two methods are combined to provide valuable insights into the generation of predictions and analysis in multiple applications.
  arXiv  Detail & Related papers  (2023-05-17T07:48:54Z)
• Generalizing Bayesian Optimization with Decision-theoretic Entropies [102.82152945324381]
  We consider a generalization of Shannon entropy from work in statistical decision theory. We first show that special cases of this entropy lead to popular acquisition functions used in BO procedures. We then show how alternative choices for the loss yield a flexible family of acquisition functions.
  arXiv  Detail & Related papers  (2022-10-04T04:43:58Z)
• Pre-training helps Bayesian optimization too [49.28382118032923]
  We seek an alternative practice for setting functional priors. In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori. Our results show that our method is able to locate good hyper parameters at least 3 times more efficiently than the best competing methods.
  arXiv  Detail & Related papers  (2022-07-07T04:42:54Z)
• Optimal Algorithms for Mean Estimation under Local Differential Privacy [55.32262879188817]
  We show that PrivUnit achieves the optimal variance among a large family of locally private randomizers. We also develop a new variant of PrivUnit based on the Gaussian distribution which is more amenable to mathematical analysis and enjoys the same optimality guarantees.
  arXiv  Detail & Related papers  (2022-05-05T06:43:46Z)
• Pre-trained Gaussian Processes for Bayesian Optimization [24.730678780782647]
  We propose a new pre-training based BO framework named HyperBO. We show bounded posterior predictions and near-zero regrets for HyperBO without assuming the "ground truth" GP prior is known.
  arXiv  Detail & Related papers  (2021-09-16T20:46:26Z)
• Bayesian Variational Optimization for Combinatorial Spaces [0.0]
  Broad applications include the study of molecules, proteins, DNA, device structures and quantum circuit designs. A on optimization over categorical spaces is needed to find optimal or pareto-optimal solutions. We introduce a variational Bayesian optimization method that combines variational optimization and continuous relaxations.
  arXiv  Detail & Related papers  (2020-11-03T20:56:13Z)
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization [4.191847852775072]
  We present an alternative objective that is equivalent to a Probably Approximately Correct-Bayes (PAC-Bayes) bound on the expected out-of-sample error. We then devise an efficient gradient-based algorithm to minimize this objective.
  arXiv  Detail & Related papers  (2020-08-14T15:54:51Z)
• Incorporating Expert Prior in Bayesian Optimisation via Space Warping [54.412024556499254]
  In big search spaces the algorithm goes through several low function value regions before reaching the optimum of the function. One approach to subside this cold start phase is to use prior knowledge that can accelerate the optimisation. In this paper, we represent the prior knowledge about the function optimum through a prior distribution. The prior distribution is then used to warp the search space in such a way that space gets expanded around the high probability region of function optimum and shrinks around low probability region of optimum.
  arXiv  Detail & Related papers  (2020-03-27T06:18:49Z)
• Towards Automatic Bayesian Optimization: A first step involving acquisition functions [0.0]
  Bayesian optimization is the state of the art technique for the optimization of black boxes, i.e., functions where we do not have access to their analytical expression. We propose a first attempt over automatic bayesian optimization by exploring several techniques that automatically tune the acquisition function.
  arXiv  Detail & Related papers  (2020-03-21T12:22:45Z)
• Implicit differentiation of Lasso-type models for hyperparameter optimization [82.73138686390514]
  We introduce an efficient implicit differentiation algorithm, without matrix inversion, tailored for Lasso-type problems. Our approach scales to high-dimensional data by leveraging the sparsity of the solutions.
  arXiv  Detail & Related papers  (2020-02-20T18:43:42Z)
• Bilevel Optimization for Differentially Private Optimization in Energy Systems [53.806512366696275]
  This paper studies how to apply differential privacy to constrained optimization problems whose inputs are sensitive. The paper shows that, under a natural assumption, a bilevel model can be solved efficiently for large-scale nonlinear optimization problems.
  arXiv  Detail & Related papers  (2020-01-26T20:15:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.