A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization

• URL: http://arxiv.org/abs/2212.04486v3
• Date: Sun, 5 May 2024 20:26:16 GMT
• Title: A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization
• Authors: Ashwinee Panda, Xinyu Tang, Saeed Mahloujifar, Vikash Sehwag, Prateek Mittal,
• Abstract summary: We propose an adaptive HPO method to account for the privacy cost of HPO. We obtain state-of-the-art performance on 22 benchmark tasks, across computer vision and natural language processing, across pretraining and finetuning.
• Score: 57.450449884166346
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: An open problem in differentially private deep learning is hyperparameter optimization (HPO). DP-SGD introduces new hyperparameters and complicates existing ones, forcing researchers to painstakingly tune hyperparameters with hundreds of trials, which in turn makes it impossible to account for the privacy cost of HPO without destroying the utility. We propose an adaptive HPO method that uses cheap trials (in terms of privacy cost and runtime) to estimate optimal hyperparameters and scales them up. We obtain state-of-the-art performance on 22 benchmark tasks, across computer vision and natural language processing, across pretraining and finetuning, across architectures and a wide range of $\varepsilon \in [0.01,8.0]$, all while accounting for the privacy cost of HPO.

Related papers

• PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning [49.92394599459274]
  We propose PriorBand, an HPO algorithm tailored to Deep Learning (DL) pipelines. We show its robustness across a range of DL benchmarks and show its gains under informative expert input and against poor expert beliefs.
  arXiv  Detail & Related papers  (2023-06-21T16:26:14Z)
• DP-HyPO: An Adaptive Private Hyperparameter Optimization Framework [31.628466186344582]
  We introduce DP-HyPO, a pioneering framework for adaptive'' private hyperparameter optimization. We provide a comprehensive differential privacy analysis of our framework. We empirically demonstrate the effectiveness of DP-HyPO on a diverse set of real-world datasets.
  arXiv  Detail & Related papers  (2023-06-09T07:55:46Z)
• Practical Differentially Private Hyperparameter Tuning with Subsampling [8.022555128083026]
  We propose a new class of differentially private (DP) machine learning (ML) algorithms, where the number of random search samples is randomized itself. We focus on lowering both the DP bounds and the computational cost of these methods by using only a random subset of the sensitive data. We provide a R'enyi differential privacy analysis for the proposed method and experimentally show that it consistently leads to better privacy-utility trade-off.
  arXiv  Detail & Related papers  (2023-01-27T21:01:58Z)
• TAN Without a Burn: Scaling Laws of DP-SGD [70.7364032297978]
  Differentially Private methods for training Deep Neural Networks (DNNs) have progressed recently. We decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements. We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a +9 points gain in top-1 accuracy.
  arXiv  Detail & Related papers  (2022-10-07T08:44:35Z)
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning [72.54359545547904]
  We propose a gradient-based subset selection framework for hyper- parameter tuning. We show that using gradient-based data subsets for hyper- parameter tuning achieves significantly faster turnaround times and speedups of 3$times$-30$times$.
  arXiv  Detail & Related papers  (2022-03-15T19:25:01Z)
• The Role of Adaptive Optimizers for Honest Private Hyperparameter Selection [12.38071940409141]
  We show that standard composition tools outperform more advanced techniques in many settings. We draw upon limiting behaviour of Adam in the DP setting to design a new and more efficient tool.
  arXiv  Detail & Related papers  (2021-11-09T01:56:56Z)
• Cost-Efficient Online Hyperparameter Optimization [94.60924644778558]
  We propose an online HPO algorithm that reaches human expert-level performance within a single run of the experiment. Our proposed online HPO algorithm reaches human expert-level performance within a single run of the experiment, while incurring only modest computational overhead compared to regular training.
  arXiv  Detail & Related papers  (2021-01-17T04:55:30Z)
• HyperSTAR: Task-Aware Hyperparameters for Deep Networks [52.50861379908611]
  HyperSTAR is a task-aware method to warm-start HPO for deep neural networks. It learns a dataset (task) representation along with the performance predictor directly from raw images. It evaluates 50% less configurations to achieve the best performance compared to existing methods.
  arXiv  Detail & Related papers  (2020-05-21T08:56:50Z)
• Frugal Optimization for Cost-related Hyperparameters [43.599155206275306]
  We develop a new cost-frugal HPO solution for machine learning algorithms. We prove a convergence rate of $O(fracsqrtdsqrtK)$ and an $O(depsilon-2)$-approximation guarantee on the total cost. We provide strong empirical results in comparison with state-of-the-art HPO methods on large AutoML benchmarks.
  arXiv  Detail & Related papers  (2020-05-04T15:40:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.