Hyperparameter Importance Analysis for Multi-Objective AutoML

• URL: http://arxiv.org/abs/2405.07640v3
• Date: Thu, 02 Jan 2025 13:46:53 GMT
• Title: Hyperparameter Importance Analysis for Multi-Objective AutoML
• Authors: Daphne Theodorakopoulos, Frederic Stahl, Marius Lindauer,
• Abstract summary: In this paper, we propose the first method for assessing the importance of hyper parameters in multi-objective optimization tasks. Specifically, we compute the a-priori scalarization of the objectives and determine the importance of the hyper parameters for different objective tradeoffs.
• Score: 14.336028105614824
• License:
• Abstract: Hyperparameter optimization plays a pivotal role in enhancing the predictive performance and generalization capabilities of ML models. However, in many applications, we do not only care about predictive performance but also about additional objectives such as inference time, memory, or energy consumption. In such multi-objective scenarios, determining the importance of hyperparameters poses a significant challenge due to the complex interplay between the conflicting objectives. In this paper, we propose the first method for assessing the importance of hyperparameters in multi-objective hyperparameter optimization. Our approach leverages surrogate-based hyperparameter importance measures, i.e., fANOVA and ablation paths, to provide insights into the impact of hyperparameters on the optimization objectives. Specifically, we compute the a-priori scalarization of the objectives and determine the importance of the hyperparameters for different objective tradeoffs. Through extensive empirical evaluations on diverse benchmark datasets with three different objective pairs, each combined with accuracy, namely time, demographic parity loss, and energy consumption, we demonstrate the effectiveness and robustness of our proposed method. Our findings not only offer valuable guidance for hyperparameter tuning in multi-objective optimization tasks but also contribute to advancing the understanding of hyperparameter importance in complex optimization scenarios.

Related papers

• Efficient Hyperparameter Importance Assessment for CNNs [1.7778609937758323]
  This paper aims to quantify the importance weights of some hyperparameters in Convolutional Neural Networks (CNNs) with an algorithm called N-RReliefF. We conduct an extensive study by training over ten thousand CNN models across ten popular image classification datasets.
  arXiv  Detail & Related papers  (2024-10-11T15:47:46Z)
• Scaling Exponents Across Parameterizations and Optimizers [94.54718325264218]
  We propose a new perspective on parameterization by investigating a key assumption in prior work. Our empirical investigation includes tens of thousands of models trained with all combinations of threes. We find that the best learning rate scaling prescription would often have been excluded by the assumptions in prior work.
  arXiv  Detail & Related papers  (2024-07-08T12:32:51Z)
• ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections [59.839926875976225]
  We propose the ETHER transformation family, which performs Efficient fineTuning via HypErplane Reflections. In particular, we introduce ETHER and its relaxation ETHER+, which match or outperform existing PEFT methods with significantly fewer parameters.
  arXiv  Detail & Related papers  (2024-05-30T17:26:02Z)
• Trajectory-Based Multi-Objective Hyperparameter Optimization for Model Retraining [8.598456741786801]
  We present a novel trajectory-based multi-objective Bayesian optimization algorithm. Our algorithm outperforms the state-of-the-art multi-objectives in both locating better trade-offs and tuning efficiency.
  arXiv  Detail & Related papers  (2024-05-24T07:43:45Z)
• End-to-End Learning for Fair Multiobjective Optimization Under Uncertainty [55.04219793298687]
  The Predict-Then-Forecast (PtO) paradigm in machine learning aims to maximize downstream decision quality. This paper extends the PtO methodology to optimization problems with nondifferentiable Ordered Weighted Averaging (OWA) objectives. It shows how optimization of OWA functions can be effectively integrated with parametric prediction for fair and robust optimization under uncertainty.
  arXiv  Detail & Related papers  (2024-02-12T16:33:35Z)
• Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectives [5.94867851915494]
  We propose a multi-objective Bayesian optimization (MoBO) algorithm that addresses these problems. We increase the efficiency of our approach by imposing constraints on the objective to avoid exploring unnecessary configurations. Finally, we leverage an approach to parallelize the MoBO which results in a 5x speed-up when using 16x more workers.
  arXiv  Detail & Related papers  (2023-09-26T13:48:04Z)
• Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning [65.51668094117802]
  We propose a human-centered interactive HPO approach tailored towards multi-objective machine learning (ML) Instead of relying on the user guessing the most suitable indicator for their needs, our approach automatically learns an appropriate indicator.
  arXiv  Detail & Related papers  (2023-09-07T09:22:05Z)
• Multi-objective hyperparameter optimization with performance uncertainty [62.997667081978825]
  This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of Machine Learning algorithms. We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise. Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR.
  arXiv  Detail & Related papers  (2022-09-09T14:58:43Z)
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning [72.54359545547904]
  We propose a gradient-based subset selection framework for hyper- parameter tuning. We show that using gradient-based data subsets for hyper- parameter tuning achieves significantly faster turnaround times and speedups of 3$times$-30$times$.
  arXiv  Detail & Related papers  (2022-03-15T19:25:01Z)
• Efficient Hyperparameter Optimization under Multi-Source Covariate Shift [13.787554178089446]
  A typical assumption in supervised machine learning is that the train (source) and test (target) datasets follow completely the same distribution. In this work, we consider a novel hyperparameter optimization problem under the multi-source covariate shift. We construct a variance reduced estimator that unbiasedly approximates the target objective with a desirable variance property.
  arXiv  Detail & Related papers  (2020-06-18T15:10:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.