On Noisy Evaluation in Federated Hyperparameter Tuning

• URL: http://arxiv.org/abs/2212.08930v4
• Date: Mon, 15 May 2023 16:44:06 GMT
• Title: On Noisy Evaluation in Federated Hyperparameter Tuning
• Abstract summary: Small amounts of noise can significantly impact tuning methods. We propose a simple and effective approach that leverages public proxy data to boost the evaluation signal.
• Score: 27.97637333107678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hyperparameter tuning is critical to the success of federated learning applications. Unfortunately, appropriately selecting hyperparameters is challenging in federated networks. Issues of scale, privacy, and heterogeneity introduce noise in the tuning process and make it difficult to evaluate the performance of various hyperparameters. In this work, we perform the first systematic study on the effect of noisy evaluation in federated hyperparameter tuning. We first identify and rigorously explore key sources of noise, including client subsampling, data and systems heterogeneity, and data privacy. Surprisingly, our results indicate that even small amounts of noise can significantly impact tuning methods-reducing the performance of state-of-the-art approaches to that of naive baselines. To address noisy evaluation in such scenarios, we propose a simple and effective approach that leverages public proxy data to boost the evaluation signal. Our work establishes general challenges, baselines, and best practices for future work in federated hyperparameter tuning.

Related papers

• Addressing prior dependence in hierarchical Bayesian modeling for PTA data analysis II: Noise and SGWB inference through parameter decorrelation [0.0]
  PTA analyses assign fixed uniform noise priors to each pulsar, an approach that can introduce systematic biases when combining the array.<n>We adopt a hierarchical Bayesian modeling strategy in which noise priors are parametrized by higher-level hyper parameters.<n>We show that the hierarchical treatment constrains the noise parameters more tightly and partially alleviates the red-noise-SGWB degeneracy.
  arXiv  Detail & Related papers  (2025-11-03T17:54:55Z)
• One-step Noisy Label Mitigation [86.57572253460125]
  Mitigating the detrimental effects of noisy labels on the training process has become increasingly critical. We propose One-step Anti-Noise (OSA), a model-agnostic noisy label mitigation paradigm. We empirically demonstrate the superiority of OSA, highlighting its enhanced training robustness, improved task transferability, ease of deployment, and reduced computational costs.
  arXiv  Detail & Related papers  (2024-10-02T18:42:56Z)
• ROPO: Robust Preference Optimization for Large Language Models [59.10763211091664]
  We propose an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models. Experiments on three widely-used datasets with Mistral-7B and Llama-2-7B demonstrate that ROPO significantly outperforms existing preference alignment methods.
  arXiv  Detail & Related papers  (2024-04-05T13:58:51Z)
• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Adaptive Differential Privacy in Federated Learning: A Priority-Based Approach [0.0]
  Federated learning (FL) develops global models without direct access to local datasets. DP offers a framework that gives a privacy guarantee by adding certain amounts of noise to parameters. We propose adaptive noise addition in FL which decides the value of injected noise based on features' relative importance.
  arXiv  Detail & Related papers  (2024-01-04T03:01:15Z)
• Improve Noise Tolerance of Robust Loss via Noise-Awareness [60.34670515595074]
  We propose a meta-learning method which is capable of adaptively learning a hyper parameter prediction function, called Noise-Aware-Robust-Loss-Adjuster (NARL-Adjuster for brevity) Four SOTA robust loss functions are attempted to be integrated with our algorithm, and comprehensive experiments substantiate the general availability and effectiveness of the proposed method in both its noise tolerance and performance.
  arXiv  Detail & Related papers  (2023-01-18T04:54:58Z)
• Exploring Robustness of Prefix Tuning in Noisy Data: A Case Study in Financial Sentiment Analysis [0.0]
  We show that fine-tuning is more robust to noise than prefix tuning when facing noisy data. We strongly advocate that caution should be carefully taken when applying the state-of-the-art prefix tuning method to noisy data.
  arXiv  Detail & Related papers  (2022-10-26T01:13:41Z)
• Partial Identification with Noisy Covariates: A Robust Optimization Approach [94.10051154390237]
  Causal inference from observational datasets often relies on measuring and adjusting for covariates. We show that this robust optimization approach can extend a wide range of causal adjustment methods to perform partial identification. Across synthetic and real datasets, we find that this approach provides ATE bounds with a higher coverage probability than existing methods.
  arXiv  Detail & Related papers  (2022-02-22T04:24:26Z)
• Efficient Hyperparameter Tuning for Large Scale Kernel Ridge Regression [19.401624974011746]
  We propose a complexity regularization criterion based on a data dependent penalty, and discuss its efficient optimization. Our analysis shows the benefit of the proposed approach, that we hence incorporate in a library for large scale kernel methods to derive adaptively tuned solutions.
  arXiv  Detail & Related papers  (2022-01-17T09:57:32Z)
• Amortized Auto-Tuning: Cost-Efficient Transfer Optimization for Hyperparameter Recommendation [83.85021205445662]
  We propose an instantiation--amortized auto-tuning (AT2) to speed up tuning of machine learning models. We conduct a thorough analysis of the multi-task multi-fidelity Bayesian optimization framework, which leads to the best instantiation--amortized auto-tuning (AT2)
  arXiv  Detail & Related papers  (2021-06-17T00:01:18Z)
• Tuning-free Plug-and-Play Proximal Algorithm for Inverse Imaging Problems [22.239477171296056]
  Plug-and-play () is a non customize framework that combines different learned algorithms with advanced denoiser priors. A key problem. based approaches is that they require manual parameter tweaking. A key part of our approach is develop a network policy for automatic tuning of parameters. We show promising results on Compressed Sensing and phase retrieval.
  arXiv  Detail & Related papers  (2020-02-22T03:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.