Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization

• URL: http://arxiv.org/abs/2601.03166v1
• Date: Tue, 06 Jan 2026 16:37:44 GMT
• Title: Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization
• Authors: Daphne Theodorakopoulos, Marcel Wever, Marius Lindauer,
• Abstract summary: Multi-objective optimization (MOO) is used to trade off objectives in choosing a suitable ML model.<n>We propose a novel dynamic optimization approach that prioritizes the most influential hyperparameters based on varying objective trade-offs.
• Score: 10.530794046739619
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Choosing a suitable ML model is a complex task that can depend on several objectives, e.g., accuracy, model size, fairness, inference time, or energy consumption. In practice, this requires trading off multiple, often competing, objectives through multi-objective optimization (MOO). However, existing MOO methods typically treat all hyperparameters as equally important, overlooking that hyperparameter importance (HPI) can vary significantly depending on the trade-off between objectives. We propose a novel dynamic optimization approach that prioritizes the most influential hyperparameters based on varying objective trade-offs during the search process, which accelerates empirical convergence and leads to better solutions. Building on prior work on HPI for MOO post-analysis, we now integrate HPI, calculated with HyperSHAP, into the optimization. For this, we leverage the objective weightings naturally produced by the MOO algorithm ParEGO and adapt the configuration space by fixing the unimportant hyperparameters, allowing the search to focus on the important ones. Eventually, we validate our method with diverse tasks from PyMOO and YAHPO-Gym. Empirical results demonstrate improvements in convergence speed and Pareto front quality compared to baselines.

Related papers

• Parametric Pareto Set Learning for Expensive Multi-Objective Optimization [16.780031024741223]
  Parametric multi-objective optimization (PMO) addresses the challenge of solving an infinite family of multi-objective optimization problems.<n>Traditional methods require re-execution for each parameter configuration, leading to prohibitive costs when objective evaluations are computationally expensive.
  arXiv  Detail & Related papers  (2025-11-08T03:05:28Z)
• Hyperparameter Importance Analysis for Multi-Objective AutoML [14.336028105614824]
  In this paper, we propose the first method for assessing the importance of hyper parameters in multi-objective optimization tasks.<n>Specifically, we compute the a-priori scalarization of the objectives and determine the importance of the hyper parameters for different objective tradeoffs.
  arXiv  Detail & Related papers  (2024-05-13T11:00:25Z)
• End-to-End Learning for Fair Multiobjective Optimization Under Uncertainty [55.04219793298687]
  The Predict-Then-Forecast (PtO) paradigm in machine learning aims to maximize downstream decision quality. This paper extends the PtO methodology to optimization problems with nondifferentiable Ordered Weighted Averaging (OWA) objectives. It shows how optimization of OWA functions can be effectively integrated with parametric prediction for fair and robust optimization under uncertainty.
  arXiv  Detail & Related papers  (2024-02-12T16:33:35Z)
• Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectives [5.94867851915494]
  We propose a multi-objective Bayesian optimization (MoBO) algorithm that addresses these problems. We increase the efficiency of our approach by imposing constraints on the objective to avoid exploring unnecessary configurations. Finally, we leverage an approach to parallelize the MoBO which results in a 5x speed-up when using 16x more workers.
  arXiv  Detail & Related papers  (2023-09-26T13:48:04Z)
• Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning [65.51668094117802]
  We propose a human-centered interactive HPO approach tailored towards multi-objective machine learning (ML) Instead of relying on the user guessing the most suitable indicator for their needs, our approach automatically learns an appropriate indicator.
  arXiv  Detail & Related papers  (2023-09-07T09:22:05Z)
• HomOpt: A Homotopy-Based Hyperparameter Optimization Method [10.11271414863925]
  We propose HomOpt, a data-driven approach based on a generalized additive model (GAM) surrogate combined with homotopy optimization. We show how HomOpt can boost the performance and effectiveness of any given method with faster convergence to the optimum on continuous discrete, and categorical domain spaces.
  arXiv  Detail & Related papers  (2023-08-07T06:01:50Z)
• BOtied: Multi-objective Bayesian optimization with tied multivariate ranks [33.414682601242006]
  In this paper, we show a natural connection between non-dominated solutions and the extreme quantile of the joint cumulative distribution function. Motivated by this link, we propose the Pareto-compliant CDF indicator and the associated acquisition function, BOtied. Our experiments on a variety of synthetic and real-world problems demonstrate that BOtied outperforms state-of-the-art MOBO acquisition functions.
  arXiv  Detail & Related papers  (2023-06-01T04:50:06Z)
• Multi-objective hyperparameter optimization with performance uncertainty [62.997667081978825]
  This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of Machine Learning algorithms. We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise. Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR.
  arXiv  Detail & Related papers  (2022-09-09T14:58:43Z)
• Pre-training helps Bayesian optimization too [49.28382118032923]
  We seek an alternative practice for setting functional priors. In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori. Our results show that our method is able to locate good hyper parameters at least 3 times more efficiently than the best competing methods.
  arXiv  Detail & Related papers  (2022-07-07T04:42:54Z)
• Consolidated learning -- a domain-specific model-free optimization strategy with examples for XGBoost and MIMIC-IV [4.370097023410272]
  This paper proposes a new formulation of the tuning problem, called consolidated learning. In such settings, we are interested in the total optimization time rather than tuning for a single task. We demonstrate the effectiveness of this approach through an empirical study for XGBoost algorithm and the collection of predictive tasks extracted from the MIMIC-IV medical database.
  arXiv  Detail & Related papers  (2022-01-27T21:38:53Z)
• A survey on multi-objective hyperparameter optimization algorithms for Machine Learning [62.997667081978825]
  This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms. We distinguish between metaheuristic-based algorithms, metamodel-based algorithms, and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.
  arXiv  Detail & Related papers  (2021-11-23T10:22:30Z)
• Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm [97.66038345864095]
  We propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG) Specifically, we first formulate hyperparameter optimization as an A-based constrained optimization problem. Then, we use the average zeroth-order hyper-gradients to update hyper parameters.
  arXiv  Detail & Related papers  (2021-02-17T21:03:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.