SELECTOR: Selecting a Representative Benchmark Suite for Reproducible Statistical Comparison

• URL: http://arxiv.org/abs/2204.11527v1
• Date: Mon, 25 Apr 2022 09:38:43 GMT
• Title: SELECTOR: Selecting a Representative Benchmark Suite for Reproducible Statistical Comparison
• Authors: Gjorgjina Cenikj, Ryan Dieter Lang, Andries Petrus Engelbrecht, Carola Doerr, Peter Koro\v{s}ec and Tome Eftimov
• Abstract summary: We evaluate three approaches for selecting diverse problem instances which should be involved in the comparison of optimization algorithms. The first approach employs clustering to identify similar groups of problem instances and subsequent sampling from each cluster to construct new benchmarks. The other two approaches use graph algorithms for identifying dominating and maximal independent sets of nodes.
• Score: 0.7046417074932257
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fair algorithm evaluation is conditioned on the existence of high-quality benchmark datasets that are non-redundant and are representative of typical optimization scenarios. In this paper, we evaluate three heuristics for selecting diverse problem instances which should be involved in the comparison of optimization algorithms in order to ensure robust statistical algorithm performance analysis. The first approach employs clustering to identify similar groups of problem instances and subsequent sampling from each cluster to construct new benchmarks, while the other two approaches use graph algorithms for identifying dominating and maximal independent sets of nodes. We demonstrate the applicability of the proposed heuristics by performing a statistical performance analysis of five portfolios consisting of three optimization algorithms on five of the most commonly used optimization benchmarks. The results indicate that the statistical analyses of the algorithms' performance, conducted on each benchmark separately, produce conflicting outcomes, which can be used to give a false indication of the superiority of one algorithm over another. On the other hand, when the analysis is conducted on the problem instances selected with the proposed heuristics, which uniformly cover the problem landscape, the statistical outcomes are robust and consistent.

Related papers

• HRA: A Multi-Criteria Framework for Ranking Metaheuristic Optimization Algorithms [0.0]
  The HRA algorithm aims to efficiently rank metaheuristic algorithms based on their performance across many criteria and dimensions. Our study uses data from the CEC 2017 competition to demonstrate the robustness and efficacy of the HRA framework.
  arXiv  Detail & Related papers  (2024-09-18T00:44:50Z)
• Generalization Bounds of Surrogate Policies for Combinatorial Optimization Problems [61.580419063416734]
  A recent stream of structured learning approaches has improved the practical state of the art for a range of optimization problems. The key idea is to exploit the statistical distribution over instances instead of dealing with instances separately. In this article, we investigate methods that smooth the risk by perturbing the policy, which eases optimization and improves the generalization error.
  arXiv  Detail & Related papers  (2024-07-24T12:00:30Z)
• Towards Robust Benchmarking of Quantum Optimization Algorithms [3.9456729020535013]
  A key problem in existing benchmarking frameworks is the lack of equal effort in optimizing for the best quantum and, respectively, classical approaches. This paper presents a comprehensive set of guidelines comprising universal steps towards fair benchmarks.
  arXiv  Detail & Related papers  (2024-05-13T10:35:23Z)
• Quantized Hierarchical Federated Learning: A Robust Approach to Statistical Heterogeneity [3.8798345704175534]
  We present a novel hierarchical federated learning algorithm that incorporates quantization for communication-efficiency. We offer a comprehensive analytical framework to evaluate its optimality gap and convergence rate. Our findings reveal that our algorithm consistently achieves high learning accuracy over a range of parameters.
  arXiv  Detail & Related papers  (2024-03-03T15:40:24Z)
• Prasatul Matrix: A Direct Comparison Approach for Analyzing Evolutionary Optimization Algorithms [2.1320960069210475]
  Direct comparison approach is proposed to analyze the performance of evolutionary optimization algorithms. Five different performance measures are designed based on the prasatul matrix to evaluate the performance of algorithms.
  arXiv  Detail & Related papers  (2022-12-01T17:21:44Z)
• Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability [107.65337427333064]
  optimization of the Area Under the Precision-Recall Curve (AUPRC) is a crucial problem for machine learning. In this work, we present the first trial in the single-dependent generalization of AUPRC optimization. Experiments on three image retrieval datasets on speak to the effectiveness and soundness of our framework.
  arXiv  Detail & Related papers  (2022-09-27T09:06:37Z)
• Estimating leverage scores via rank revealing methods and randomization [50.591267188664666]
  We study algorithms for estimating the statistical leverage scores of rectangular dense or sparse matrices of arbitrary rank. Our approach is based on combining rank revealing methods with compositions of dense and sparse randomized dimensionality reduction transforms.
  arXiv  Detail & Related papers  (2021-05-23T19:21:55Z)
• On the Assessment of Benchmark Suites for Algorithm Comparison [7.501426386641256]
  We show that most benchmark functions of BBOB suite have high difficulty levels (compared to the optimization algorithms) and low discrimination. We discuss potential uses of IRT in benchmarking, including its use to improve the design of benchmark suites.
  arXiv  Detail & Related papers  (2021-04-15T11:20:11Z)
• Bilevel Optimization: Convergence Analysis and Enhanced Design [63.64636047748605]
  Bilevel optimization is a tool for many machine learning problems. We propose a novel stoc-efficientgradient estimator named stoc-BiO.
  arXiv  Detail & Related papers  (2020-10-15T18:09:48Z)
• Ranking a set of objects: a graph based least-square approach [70.7866286425868]
  We consider the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of equal workers. We propose a class of non-adaptive ranking algorithms that rely on a least-squares intrinsic optimization criterion for the estimation of qualities.
  arXiv  Detail & Related papers  (2020-02-26T16:19:09Z)
• Extreme Algorithm Selection With Dyadic Feature Representation [78.13985819417974]
  We propose the setting of extreme algorithm selection (XAS) where we consider fixed sets of thousands of candidate algorithms. We assess the applicability of state-of-the-art AS techniques to the XAS setting and propose approaches leveraging a dyadic feature representation.
  arXiv  Detail & Related papers  (2020-01-29T09:40:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.