FEATHERS: Federated Architecture and Hyperparameter Search

• URL: http://arxiv.org/abs/2206.12342v3
• Date: Mon, 27 Mar 2023 09:26:59 GMT
• Title: FEATHERS: Federated Architecture and Hyperparameter Search
• Authors: Jonas Seng, Pooja Prasad, Martin Mundt, Devendra Singh Dhami, Kristian Kersting
• Abstract summary: We introduce FEATHERS - $textbfFE$derated $textbfA$rchi$textbfT$ecture and $textbfH$yp$textbfER$ $textbfS$earch.
• Score: 18.217609851581084
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural architectures have profound impact on achieved performance in many of today's AI tasks, yet, their design still heavily relies on human prior knowledge and experience. Neural architecture search (NAS) together with hyperparameter optimization (HO) helps to reduce this dependence. However, state of the art NAS and HO rapidly become infeasible with increasing amount of data being stored in a distributed fashion, typically violating data privacy regulations such as GDPR and CCPA. As a remedy, we introduce FEATHERS - $\textbf{FE}$derated $\textbf{A}$rchi$\textbf{T}$ecture and $\textbf{H}$yp$\textbf{ER}$parameter $\textbf{S}$earch, a method that not only optimizes both neural architectures and optimization-related hyperparameters jointly in distributed data settings, but further adheres to data privacy through the use of differential privacy (DP). We show that FEATHERS efficiently optimizes architectural and optimization-related hyperparameters alike, while demonstrating convergence on classification tasks at no detriment to model performance when complying with privacy constraints.

Related papers

• ZeroLM: Data-Free Transformer Architecture Search for Language Models [54.83882149157548]
  Current automated proxy discovery approaches suffer from extended search times, susceptibility to data overfitting, and structural complexity.<n>This paper introduces a novel zero-cost proxy methodology that quantifies model capacity through efficient weight statistics.<n>Our evaluation demonstrates the superiority of this approach, achieving a Spearman's rho of 0.76 and Kendall's tau of 0.53 on the FlexiBERT benchmark.
  arXiv  Detail & Related papers  (2025-03-24T13:11:22Z)
• Towards hyperparameter-free optimization with differential privacy [9.193537596304669]
  Differential privacy (DP) is a privacy-preserving paradigm that protects the training data when training deep learning models. In this work, we adapt the automatic learning rate schedule to DP optimization for any models and achieves state-of-the-art DP performance on various language and vision tasks.
  arXiv  Detail & Related papers  (2025-03-02T02:59:52Z)
• Efficient Hyperparameter Importance Assessment for CNNs [1.7778609937758323]
  This paper aims to quantify the importance weights of some hyperparameters in Convolutional Neural Networks (CNNs) with an algorithm called N-RReliefF. We conduct an extensive study by training over ten thousand CNN models across ten popular image classification datasets.
  arXiv  Detail & Related papers  (2024-10-11T15:47:46Z)
• POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator [4.09225917049674]
  Transferable NAS has emerged, generalizing the search process from dataset-dependent to task-dependent. This paper introduces POMONAG, extending DiffusionNAG via a many-optimal diffusion process. Results were validated on two search spaces -- NAS201 and MobileNetV3 -- and evaluated across 15 image classification datasets.
  arXiv  Detail & Related papers  (2024-09-30T16:05:29Z)
• Differentially Private Fine-Tuning of Diffusion Models [22.454127503937883]
  The integration of Differential Privacy with diffusion models (DMs) presents a promising yet challenging frontier. Recent developments in this field have highlighted the potential for generating high-quality synthetic data by pre-training on public data. We propose a strategy optimized for private diffusion models, which minimizes the number of trainable parameters to enhance the privacy-utility trade-off.
  arXiv  Detail & Related papers  (2024-06-03T14:18:04Z)
• Efficient Architecture Search via Bi-level Data Pruning [70.29970746807882]
  This work pioneers an exploration into the critical role of dataset characteristics for DARTS bi-level optimization. We introduce a new progressive data pruning strategy that utilizes supernet prediction dynamics as the metric. Comprehensive evaluations on the NAS-Bench-201 search space, DARTS search space, and MobileNet-like search space validate that BDP reduces search costs by over 50%.
  arXiv  Detail & Related papers  (2023-12-21T02:48:44Z)
• Fairer and More Accurate Tabular Models Through NAS [14.147928131445852]
  We propose using multi-objective Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) in the first application to the very challenging domain of tabular data. We show that models optimized solely for accuracy with NAS often fail to inherently address fairness concerns. We produce architectures that consistently dominate state-of-the-art bias mitigation methods either in fairness, accuracy or both.
  arXiv  Detail & Related papers  (2023-10-18T17:56:24Z)
• Efficient and Joint Hyperparameter and Architecture Search for Collaborative Filtering [31.25094171513831]
  We propose a two-stage search algorithm for Collaborative Filtering models. In the first stage, we leverage knowledge from subsampled datasets to reduce evaluation costs. In the second stage, we efficiently fine-tune top candidate models on the whole dataset.
  arXiv  Detail & Related papers  (2023-07-12T10:56:25Z)
• A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization [57.450449884166346]
  We propose an adaptive HPO method to account for the privacy cost of HPO. We obtain state-of-the-art performance on 22 benchmark tasks, across computer vision and natural language processing, across pretraining and finetuning.
  arXiv  Detail & Related papers  (2022-12-08T18:56:37Z)
• Pre-training helps Bayesian optimization too [49.28382118032923]
  We seek an alternative practice for setting functional priors. In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori. Our results show that our method is able to locate good hyper parameters at least 3 times more efficiently than the best competing methods.
  arXiv  Detail & Related papers  (2022-07-07T04:42:54Z)
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning [72.54359545547904]
  We propose a gradient-based subset selection framework for hyper- parameter tuning. We show that using gradient-based data subsets for hyper- parameter tuning achieves significantly faster turnaround times and speedups of 3$times$-30$times$.
  arXiv  Detail & Related papers  (2022-03-15T19:25:01Z)
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture [81.82173855071312]
  We propose an end-to-end solution that integrates the AutoML components and returns a ready-to-use model at the end of the search. Dha achieves state-of-the-art (SOTA) results on various datasets, especially 77.4% accuracy on ImageNet with cell based search space.
  arXiv  Detail & Related papers  (2021-09-13T08:12:50Z)
• iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients [75.41173109807735]
  Differentiable ARchiTecture Search (DARTS) has recently become the mainstream of neural architecture search (NAS) We tackle the hypergradient computation in DARTS based on the implicit function theorem. We show that the architecture optimisation with the proposed method, named iDARTS, is expected to converge to a stationary point.
  arXiv  Detail & Related papers  (2021-06-21T00:44:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.