Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs

• URL: http://arxiv.org/abs/2501.13018v1
• Date: Wed, 22 Jan 2025 17:05:38 GMT
• Title: Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs
• Authors: Amirmohammad Farzaneh, Osvaldo Simeone,
• Abstract summary: In sensitive application domains, multi-objective hyper parameter selection can ensure the reliability of AI models prior to deployment. This paper introduces a novel framework for multi-objective hyper parameter selection using a directed acyclic graph (DAG) By integrating False Discovery Rate (FDR) control, RG-PT ensures robust statistical reliability guarantees and is shown via experiments across diverse domains.
• Score: 35.59201763567714
• License:
• Abstract: In sensitive application domains, multi-objective hyperparameter selection can ensure the reliability of AI models prior to deployment, while optimizing auxiliary performance metrics. The state-of-the-art Pareto Testing (PT) method guarantees statistical reliability constraints by adopting a multiple hypothesis testing framework. In PT, hyperparameters are validated one at a time, following a data-driven order determined by expected reliability levels. This paper introduces a novel framework for multi-objective hyperparameter selection that captures the interdependencies among the reliability levels of different hyperparameter configurations using a directed acyclic graph (DAG), which is termed the reliability graph (RG). The RG is constructed based on prior information and data by using the Bradley-Terry model. The proposed approach, RG-based PT (RG-PT), leverages the RG to enable the efficient, parallel testing of multiple hyperparameters at the same reliability level. By integrating False Discovery Rate (FDR) control, RG-PT ensures robust statistical reliability guarantees and is shown via experiments across diverse domains to consistently yield superior solutions for multi-objective calibration problems.

Related papers

• Ensuring Reliability via Hyperparameter Selection: Review and Advances [35.59201763567714]
  This paper reviews the Learn-Then-Test (LTT) framework and explores several extensions tailored to engineering-relevant scenarios. These extensions encompass different risk measures and statistical guarantees, multi-objective optimization, the incorporation of prior knowledge and dependency structures. The paper also includes illustrative applications for communication systems.
  arXiv  Detail & Related papers  (2025-02-06T16:47:21Z)
• Optimize Incompatible Parameters through Compatibility-aware Knowledge Integration [104.52015641099828]
  Existing research excels in removing such parameters or merging the outputs of multiple different pretrained models. We propose Compatibility-aware Knowledge Integration (CKI), which consists of Deep Assessment and Deep Splicing. The integrated model can be used directly for inference or for further fine-tuning.
  arXiv  Detail & Related papers  (2025-01-10T01:42:43Z)
• QADM-Net: Multi-Level Quality-Adaptive Dynamic Network for Reliable Multimodal Classification [57.08108545219043]
  Current multimodal classification methods lack dynamic networks for sample-specific depth and parameters to achieve reliable inference. We propose Multi-Level Quality-Adaptive Dynamic Multimodal Network (QADM-Net) Experiments conducted on four datasets demonstrate that QADM-Net significantly outperforms state-of-the-art methods in classification performance and reliability.
  arXiv  Detail & Related papers  (2024-12-19T03:26:51Z)
• On the consistency of hyper-parameter selection in value-based deep reinforcement learning [13.133865673667394]
  This paper conducts an empirical study focusing on the reliability of hyper- parameter selection for value-based deep reinforcement learning agents. Our findings help establish which hyper- parameters are most critical to tune, and help clarify which tunings remain consistent across different training regimes.
  arXiv  Detail & Related papers  (2024-06-25T13:06:09Z)
• High-Dimensional False Discovery Rate Control for Dependent Variables [10.86851797584794]
  We propose a dependency-aware T-Rex selector that harnesses the dependency structure among variables. We prove that our variable penalization mechanism ensures FDR control. We formulate a fully integrated optimal calibration algorithm that concurrently determines the parameters of the graphical model and the T-Rex framework.
  arXiv  Detail & Related papers  (2024-01-28T22:56:16Z)
• Efficient and Robust Bayesian Selection of Hyperparameters in Dimension Reduction for Visualization [0.0]
  We introduce an efficient and robust auto-tuning framework for hyper parameter selection in dimension reduction (DR) algorithms. Our approach enables efficient hyper parameter selection with multi-objective trade-offs and allows us to perform data-driven analysis. We evaluate our results on various synthetic and real-world datasets using multiple quality metrics.
  arXiv  Detail & Related papers  (2023-06-01T05:36:22Z)
• Efficiently Controlling Multiple Risks with Pareto Testing [34.83506056862348]
  We propose a two-stage process which combines multi-objective optimization with multiple hypothesis testing. We demonstrate the effectiveness of our approach to reliably accelerate the execution of large-scale Transformer models in natural language processing (NLP) applications.
  arXiv  Detail & Related papers  (2022-10-14T15:54:39Z)
• AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuning [72.54359545547904]
  We propose a gradient-based subset selection framework for hyper- parameter tuning. We show that using gradient-based data subsets for hyper- parameter tuning achieves significantly faster turnaround times and speedups of 3$times$-30$times$.
  arXiv  Detail & Related papers  (2022-03-15T19:25:01Z)
• Differential privacy and robust statistics in high dimensions [49.50869296871643]
  High-dimensional Propose-Test-Release (HPTR) builds upon three crucial components: the exponential mechanism, robust statistics, and the Propose-Test-Release mechanism. We show that HPTR nearly achieves the optimal sample complexity under several scenarios studied in the literature.
  arXiv  Detail & Related papers  (2021-11-12T06:36:40Z)
• Understanding Overparameterization in Generative Adversarial Networks [56.57403335510056]
  Generative Adversarial Networks (GANs) are used to train non- concave mini-max optimization problems. A theory has shown the importance of the gradient descent (GD) to globally optimal solutions. We show that in an overized GAN with a $1$-layer neural network generator and a linear discriminator, the GDA converges to a global saddle point of the underlying non- concave min-max problem.
  arXiv  Detail & Related papers  (2021-04-12T16:23:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.