Statistically Valid Hyperparameter Selection: From Tuning to Guarantees

• URL: http://arxiv.org/abs/2606.25601v1
• Date: Wed, 24 Jun 2026 09:07:26 GMT
• Title: Statistically Valid Hyperparameter Selection: From Tuning to Guarantees
• Abstract summary: Hyper parameter selection is a critical step in the deployment of modern artificial intelligence systems.<n>Hyper parameter selection is typically performed using best-effort empirical methods.<n>This book presents a unified statistical framework for reliable hyper parameter selection.
• Score: 31.006941545235396
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperparameter selection is a critical step in the deployment of modern artificial intelligence systems, given the need to tune degrees of freedom such as inference-time parameters, implementation-level settings, and thresholds driving decision rules. Despite its practical importance, hyperparameter selection is typically performed using best-effort empirical methods such as grid search or Bayesian optimization, which provide no formal statistical guarantees on reliability or safety. This monograph presents a unified statistical framework for reliable hyperparameter selection, centered on the learn-then-test (LTT) paradigm, which formulates the problem as multiple hypothesis testing over a candidate set of hyperparameters. The framework enables the selection of hyperparameters that provably satisfy application-specific reliability requirements -- such as bounds on average risk, quantile risk, or information-theoretic constraints -- with explicit, finite-sample control of error probabilities. The supporting statistical machinery, namely p-values, e-values, and concentration inequalities, is developed from first principles in a dedicated appendix.

Related papers

• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Ensuring Reliability via Hyperparameter Selection: Review and Advances [35.59201763567714]
  This paper reviews the Learn-Then-Test (LTT) framework and explores several extensions tailored to engineering-relevant scenarios.<n>These extensions encompass different risk measures and statistical guarantees, multi-objective optimization, the incorporation of prior knowledge and dependency structures.<n>The paper also includes illustrative applications for communication systems.
  arXiv  Detail & Related papers  (2025-02-06T16:47:21Z)
• Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs [35.59201763567714]
  This paper introduces reliability graph-based Pareto testing (RG-PT)<n>RG-PT maintains formal reliability guarantees in terms of false discovery rate (FDR)<n>It accounts for known relationships among hyper parameters via a directed acyclic graph.
  arXiv  Detail & Related papers  (2025-01-22T17:05:38Z)
• Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection [36.407171992845456]
  We introduce adaptive learn-then-test (aLTT) to provide finite-sample statistical guarantees on the population risk of AI models.<n>Unlike the existing learn-then-test (LTT) technique, aLTT implements sequential data-dependent multiple hypothesis testing (MHT) with early termination by leveraging e-processes.
  arXiv  Detail & Related papers  (2024-09-24T08:14:26Z)
• Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization [36.14499894307206]
  This work introduces a variant of learn-then-test (LTT) that is designed to provide statistical guarantees on quantiles of a risk measure. We illustrate the practical advantages of this approach by applying the proposed algorithm to a radio access scheduling problem.
  arXiv  Detail & Related papers  (2024-07-24T15:30:12Z)
• Information-Theoretic Safe Bayesian Optimization [59.758009422067005]
  We consider a sequential decision making task, where the goal is to optimize an unknown function without evaluating parameters that violate an unknown (safety) constraint. Most current methods rely on a discretization of the domain and cannot be directly extended to the continuous case. We propose an information-theoretic safe exploration criterion that directly exploits the GP posterior to identify the most informative safe parameters to evaluate.
  arXiv  Detail & Related papers  (2024-02-23T14:31:10Z)
• Online Constraint Tightening in Stochastic Model Predictive Control: A Regression Approach [49.056933332667114]
  No analytical solutions exist for chance-constrained optimal control problems. We propose a data-driven approach for learning the constraint-tightening parameters online during control. Our approach yields constraint-tightening parameters that tightly satisfy the chance constraints.
  arXiv  Detail & Related papers  (2023-10-04T16:22:02Z)
• Error Bounds for Kernel-Based Linear System Identification with Unknown Hyperparameters [0.38073142980733]
  kernel-based method has been successfully applied in linear system identification using stable kernel designs. From a Gaussian process, it automatically provides probabilistic error bounds for the identified models.
  arXiv  Detail & Related papers  (2023-03-17T08:52:16Z)
• Probabilities Are Not Enough: Formal Controller Synthesis for Stochastic Dynamical Models with Epistemic Uncertainty [68.00748155945047]
  Capturing uncertainty in models of complex dynamical systems is crucial to designing safe controllers. Several approaches use formal abstractions to synthesize policies that satisfy temporal specifications related to safety and reachability. Our contribution is a novel abstraction-based controller method for continuous-state models with noise, uncertain parameters, and external disturbances.
  arXiv  Detail & Related papers  (2022-10-12T07:57:03Z)
• A Theoretical Framework of Almost Hyperparameter-free Hyperparameter Selection Methods for Offline Policy Evaluation [2.741266294612776]
  offline reinforcement learning (OPE) is a core technology for data-driven decision optimization without environment simulators. We introduce a new approximate hyper parameter selection framework for OPE, which defines a notion of optimality (called selection criteria) in a quantitative and interpretable manner. We derive four AHS methods each of which has different characteristics such as convergence rate and time complexity.
  arXiv  Detail & Related papers  (2022-01-07T02:23:09Z)
• Differential privacy and robust statistics in high dimensions [49.50869296871643]
  High-dimensional Propose-Test-Release (HPTR) builds upon three crucial components: the exponential mechanism, robust statistics, and the Propose-Test-Release mechanism. We show that HPTR nearly achieves the optimal sample complexity under several scenarios studied in the literature.
  arXiv  Detail & Related papers  (2021-11-12T06:36:40Z)
• Gaussian Process Uniform Error Bounds with Unknown Hyperparameters for Safety-Critical Applications [71.23286211775084]
  We introduce robust Gaussian process uniform error bounds in settings with unknown hyper parameters. Our approach computes a confidence region in the space of hyper parameters, which enables us to obtain a probabilistic upper bound for the model error. Experiments show that the bound performs significantly better than vanilla and fully Bayesian processes.
  arXiv  Detail & Related papers  (2021-09-06T17:10:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.