Enhancing Performance and Calibration in Quantile Hyperparameter Optimization

• URL: http://arxiv.org/abs/2509.17051v1
• Date: Sun, 21 Sep 2025 12:17:06 GMT
• Title: Enhancing Performance and Calibration in Quantile Hyperparameter Optimization
• Authors: Riccardo Doyle,
• Abstract summary: Conformalized quantile regression can address these estimation weaknesses.<n>This study builds upon early work in this area.<n>Proposed algorithms are rigorously benchmarked.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bayesian hyperparameter optimization relies heavily on Gaussian Process (GP) surrogates, due to robust distributional posteriors and strong performance on limited training samples. GPs however underperform in categorical hyperparameter environments or when assumptions of normality, heteroskedasticity and symmetry are excessively challenged. Conformalized quantile regression can address these estimation weaknesses, while still providing robust calibration guarantees. This study builds upon early work in this area by addressing feedback covariate shift in sequential acquisition and integrating a wider range of surrogate architectures and acquisition functions. Proposed algorithms are rigorously benchmarked against a range of state of the art hyperparameter optimization methods (GP, TPE and SMAC). Findings identify quantile surrogate architectures and acquisition functions yielding superior performance to the current quantile literature, while validating the beneficial impact of conformalization on calibration and search performance.

Related papers

• An Adaptive KKT-Based Indicator for Convergence Assessment in Multi-Objective Optimization [0.0]
  This paper revisits an entropy-inspired convergence indicator and proposes a robust adaptive reformulation based on quantile normalization.<n>The proposed indicator preserves the stationarity-based interpretation of the original formulation while improving robustness to heterogeneous distributions of stationarity residuals.
  arXiv  Detail & Related papers  (2026-03-04T13:34:28Z)
• Continual Adaptation: Environment-Conditional Parameter Generation for Object Detection in Dynamic Scenarios [54.58186816693791]
  environments constantly change over time and space, posing significant challenges for object detectors trained based on a closed-set assumption.<n>We propose a new mechanism, converting the fine-tuning process to a specific- parameter generation.<n>In particular, we first design a dual-path LoRA-based domain-aware adapter that disentangles features into domain-invariant and domain-specific components.
  arXiv  Detail & Related papers  (2025-06-30T17:14:12Z)
• A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization [19.351804144005744]
  In deep learning, tuning parameters with conditional dependence are common in practice. New GP model accounts for the dependent structure among input variables through a new kernel function. High prediction accuracy and better optimization efficiency are observed in a series of synthetic simulations and real data applications of neural networks.
  arXiv  Detail & Related papers  (2024-01-19T21:11:32Z)
• Federated Conditional Stochastic Optimization [110.513884892319]
  Conditional optimization has found in a wide range of machine learning tasks, such as in-variant learning tasks, AUPRC, andAML. This paper proposes algorithms for distributed federated learning.
  arXiv  Detail & Related papers  (2023-10-04T01:47:37Z)
• Optimizing Hyperparameters with Conformal Quantile Regression [7.316604052864345]
  We propose to leverage conformalized quantile regression which makes minimal assumptions about the observation noise. This translates to quicker HPO convergence on empirical benchmarks.
  arXiv  Detail & Related papers  (2023-05-05T15:33:39Z)
• Sharp Calibrated Gaussian Processes [58.94710279601622]
  State-of-the-art approaches for designing calibrated models rely on inflating the Gaussian process posterior variance. We present a calibration approach that generates predictive quantiles using a computation inspired by the vanilla Gaussian process posterior variance. Our approach is shown to yield a calibrated model under reasonable assumptions.
  arXiv  Detail & Related papers  (2023-02-23T12:17:36Z)
• Optimization of Annealed Importance Sampling Hyperparameters [77.34726150561087]
  Annealed Importance Sampling (AIS) is a popular algorithm used to estimates the intractable marginal likelihood of deep generative models. We present a parameteric AIS process with flexible intermediary distributions and optimize the bridging distributions to use fewer number of steps for sampling. We assess the performance of our optimized AIS for marginal likelihood estimation of deep generative models and compare it to other estimators.
  arXiv  Detail & Related papers  (2022-09-27T07:58:25Z)
• Scalable Gaussian Process Hyperparameter Optimization via Coverage Regularization [0.0]
  We present a novel algorithm which estimates the smoothness and length-scale parameters in the Matern kernel in order to improve robustness of the resulting prediction uncertainties. We achieve improved UQ over leave-one-out likelihood while maintaining a high degree of scalability as demonstrated in numerical experiments.
  arXiv  Detail & Related papers  (2022-09-22T19:23:37Z)
• Multi-objective hyperparameter optimization with performance uncertainty [62.997667081978825]
  This paper presents results on multi-objective hyperparameter optimization with uncertainty on the evaluation of Machine Learning algorithms. We combine the sampling strategy of Tree-structured Parzen Estimators (TPE) with the metamodel obtained after training a Gaussian Process Regression (GPR) with heterogeneous noise. Experimental results on three analytical test functions and three ML problems show the improvement over multi-objective TPE and GPR.
  arXiv  Detail & Related papers  (2022-09-09T14:58:43Z)
• Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate models [0.4079265319364249]
  Current state-of-the-art methods leverage Random Forests or Gaussian processes to build a surrogate model. We propose a new surrogate model based on gradient boosting. We demonstrate empirically that the new method is able to outperform some state-of-the art techniques across a reasonable sized set of classification problems.
  arXiv  Detail & Related papers  (2021-01-06T22:07:19Z)
• Adaptive pruning-based optimization of parameterized quantum circuits [62.997667081978825]
  Variisy hybrid quantum-classical algorithms are powerful tools to maximize the use of Noisy Intermediate Scale Quantum devices. We propose a strategy for such ansatze used in variational quantum algorithms, which we call "Efficient Circuit Training" (PECT) Instead of optimizing all of the ansatz parameters at once, PECT launches a sequence of variational algorithms.
  arXiv  Detail & Related papers  (2020-10-01T18:14:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.