Large-scale Benchmarking of Metaphor-based Optimization Heuristics
- URL: http://arxiv.org/abs/2402.09800v1
- Date: Thu, 15 Feb 2024 08:54:46 GMT
- Title: Large-scale Benchmarking of Metaphor-based Optimization Heuristics
- Authors: Diederick Vermetten, Carola Doerr, Hao Wang, Anna V. Kononova, Thomas
B\"ack
- Abstract summary: We run a set of 294 algorithm implementations on the BBOB function suite.
We investigate how the choice of the budget, the performance measure, or other aspects of experimental design impact the comparison of these algorithms.
- Score: 5.081212121019668
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The number of proposed iterative optimization heuristics is growing steadily,
and with this growth, there have been many points of discussion within the
wider community. One particular criticism that is raised towards many new
algorithms is their focus on metaphors used to present the method, rather than
emphasizing their potential algorithmic contributions. Several studies into
popular metaphor-based algorithms have highlighted these problems, even
showcasing algorithms that are functionally equivalent to older existing
methods. Unfortunately, this detailed approach is not scalable to the whole set
of metaphor-based algorithms. Because of this, we investigate ways in which
benchmarking can shed light on these algorithms. To this end, we run a set of
294 algorithm implementations on the BBOB function suite. We investigate how
the choice of the budget, the performance measure, or other aspects of
experimental design impact the comparison of these algorithms. Our results
emphasize why benchmarking is a key step in expanding our understanding of the
algorithm space, and what challenges still need to be overcome to fully gauge
the potential improvements to the state-of-the-art hiding behind the metaphors.
Related papers
- A Novel Ranking Scheme for the Performance Analysis of Stochastic Optimization Algorithms using the Principles of Severity [9.310464457958844]
We provide a novel ranking scheme to rank the algorithms over multiple single-objective optimization problems.
The results of the algorithms are compared using a robust bootstrapping-based hypothesis testing procedure.
arXiv Detail & Related papers (2024-05-31T19:35:34Z) - Dual Algorithmic Reasoning [9.701208207491879]
We propose to learn algorithms by exploiting duality of the underlying algorithmic problem.
We demonstrate that simultaneously learning the dual definition of these optimisation problems in algorithmic learning allows for better learning.
We then validate the real-world utility of our dual algorithmic reasoner by deploying it on a challenging brain vessel classification task.
arXiv Detail & Related papers (2023-02-09T08:46:23Z) - Amortized Implicit Differentiation for Stochastic Bilevel Optimization [53.12363770169761]
We study a class of algorithms for solving bilevel optimization problems in both deterministic and deterministic settings.
We exploit a warm-start strategy to amortize the estimation of the exact gradient.
By using this framework, our analysis shows these algorithms to match the computational complexity of methods that have access to an unbiased estimate of the gradient.
arXiv Detail & Related papers (2021-11-29T15:10:09Z) - Algorithm Selection on a Meta Level [58.720142291102135]
We introduce the problem of meta algorithm selection, which essentially asks for the best way to combine a given set of algorithm selectors.
We present a general methodological framework for meta algorithm selection as well as several concrete learning methods as instantiations of this framework.
arXiv Detail & Related papers (2021-07-20T11:23:21Z) - Provably Faster Algorithms for Bilevel Optimization [54.83583213812667]
Bilevel optimization has been widely applied in many important machine learning applications.
We propose two new algorithms for bilevel optimization.
We show that both algorithms achieve the complexity of $mathcalO(epsilon-1.5)$, which outperforms all existing algorithms by the order of magnitude.
arXiv Detail & Related papers (2021-06-08T21:05:30Z) - Benchmarking Simulation-Based Inference [5.3898004059026325]
Recent advances in probabilistic modelling have led to a large number of simulation-based inference algorithms which do not require numerical evaluation of likelihoods.
We provide a benchmark with inference tasks and suitable performance metrics, with an initial selection of algorithms.
We found that the choice of performance metric is critical, that even state-of-the-art algorithms have substantial room for improvement, and that sequential estimation improves sample efficiency.
arXiv Detail & Related papers (2021-01-12T18:31:22Z) - A black-box adversarial attack for poisoning clustering [78.19784577498031]
We propose a black-box adversarial attack for crafting adversarial samples to test the robustness of clustering algorithms.
We show that our attacks are transferable even against supervised algorithms such as SVMs, random forests, and neural networks.
arXiv Detail & Related papers (2020-09-09T18:19:31Z) - A Brief Look at Generalization in Visual Meta-Reinforcement Learning [56.50123642237106]
We evaluate the generalization performance of meta-reinforcement learning algorithms.
We find that these algorithms can display strong overfitting when they are evaluated on challenging tasks.
arXiv Detail & Related papers (2020-06-12T15:17:17Z) - Flow-based Algorithms for Improving Clusters: A Unifying Framework,
Software, and Performance [0.0]
Clustering points in a vector space or nodes in a graph is a ubiquitous primitive in statistical data analysis.
We focus on principled algorithms for this cluster improvement problem.
We develop efficient implementations of these algorithms in our LocalGraphClustering Python package.
arXiv Detail & Related papers (2020-04-20T20:14:00Z) - Extreme Algorithm Selection With Dyadic Feature Representation [78.13985819417974]
We propose the setting of extreme algorithm selection (XAS) where we consider fixed sets of thousands of candidate algorithms.
We assess the applicability of state-of-the-art AS techniques to the XAS setting and propose approaches leveraging a dyadic feature representation.
arXiv Detail & Related papers (2020-01-29T09:40:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.