Using the Empirical Attainment Function for Analyzing Single-objective Black-box Optimization Algorithms

• URL: http://arxiv.org/abs/2404.02031v2
• Date: Sun, 15 Sep 2024 06:28:35 GMT
• Title: Using the Empirical Attainment Function for Analyzing Single-objective Black-box Optimization Algorithms
• Authors: Manuel López-Ibáñez, Diederick Vermetten, Johann Dreo, Carola Doerr,
• Abstract summary: We argue that the empirical attainment function (EAF) has several advantages over the target-based ECDF. The EAF does not require defining a priori quality targets per function, captures performance differences more precisely, and enables the use of additional summary statistics. To facilitate the accessibility of the EAF, we integrate a module to compute it into the IOHanalyzer platform.
• Score: 2.1486704308317783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A widely accepted way to assess the performance of iterative black-box optimizers is to analyze their empirical cumulative distribution function (ECDF) of pre-defined quality targets achieved not later than a given runtime. In this work, we consider an alternative approach, based on the empirical attainment function (EAF) and we show that the target-based ECDF is an approximation of the EAF. We argue that the EAF has several advantages over the target-based ECDF. In particular, it does not require defining a priori quality targets per function, captures performance differences more precisely, and enables the use of additional summary statistics that enrich the analysis. We also show that the average area over the convergence curves is a simpler-to-calculate, but equivalent, measure of anytime performance. To facilitate the accessibility of the EAF, we integrate a module to compute it into the IOHanalyzer platform. Finally, we illustrate the use of the EAF via synthetic examples and via the data available for the BBOB suite.

Related papers

• AWPO: Enhancing Tool-Use of Large Language Models through Explicit Integration of Reasoning Rewards [60.2998874976509]
  We propose advantage-weighted policy optimization (AWPO) to integrate explicit reasoning rewards to enhance tool-use capability.<n>AWPO incorporates variance-aware gating and difficulty-aware weighting to adaptively modulate advantages from reasoning signals.<n>Experiments demonstrate that AWPO achieves state-of-the-art performance across standard tool-use benchmarks.
  arXiv  Detail & Related papers  (2025-12-22T08:07:00Z)
• Rethinking Infrared Small Target Detection: A Foundation-Driven Efficient Paradigm [17.63632082331749]
  Large-scale visual foundation models (VFMs) exhibit strong generalization across diverse visual domains, but their potential for single-frame infrared small target (SIRST) detection remains largely unexplored.<n>We propose a Foundation-Driven Efficient Paradigm (FDEP) which can seamlessly adapt to existing encoder-decoder-based methods and significantly improve accuracy without additional inference overhead.
  arXiv  Detail & Related papers  (2025-12-05T08:12:35Z)
• Function Based Isolation Forest (FuBIF): A Unifying Framework for Interpretable Isolation-Based Anomaly Detection [46.53690373860206]
  Anomaly Detection (AD) is evolving through algorithms capable of identifying outliers in complex datasets.<n>This paper introduces the Isolation Forest (FuBIF), a generalization of IF that enables the use of real-valued functions for dataset branching.
  arXiv  Detail & Related papers  (2025-11-08T15:52:35Z)
• Federated Communication-Efficient Multi-Objective Optimization [27.492821176616815]
  We propose FedCMOO, a novel communication- federated multiobjective (FMOO) algorithm that improves the error convergence performance of the model compared to existing approaches. In addition, we introduce a variant of FedCMOO that allows users to specify a gradient over the objectives in terms of a desired ratio of the final objective values.
  arXiv  Detail & Related papers  (2024-10-21T18:09:22Z)
• FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch [21.41322548859776]
  We show how FunBO can be used to learn new acquisition functions written in computer code. We show how FunBO identifies AFs that generalize well in and out of the training distribution of functions.
  arXiv  Detail & Related papers  (2024-06-07T10:49:59Z)
• On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learning [1.3381647310440739]
  Potential-Based Reward Shaping (PBRS) has shown great promise in the ongoing research effort to tackle sample inefficiency in Reinforcement Learning (RL)<n>We first build some theoretically-grounded intuition on why selecting the potential function as the optimal value function of the task at hand produces performance advantages.<n>We then analyse the bias induced by finite horizons in the context of PBRS producing novel insights.
  arXiv  Detail & Related papers  (2024-04-11T15:09:49Z)
• VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness [56.87603097348203]
  VeCAF uses labels and natural language annotations to perform parametric data selection for PVM finetuning. VeCAF incorporates the finetuning objective to select significant data points that effectively guide the PVM towards faster convergence. On ImageNet, VeCAF uses up to 3.3x less training batches to reach the target performance compared to full finetuning.
  arXiv  Detail & Related papers  (2024-01-15T17:28:37Z)
• Prediction-Oriented Bayesian Active Learning [51.426960808684655]
  Expected predictive information gain (EPIG) is an acquisition function that measures information gain in the space of predictions rather than parameters. EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models.
  arXiv  Detail & Related papers  (2023-04-17T10:59:57Z)
• Unleashing the Potential of Acquisition Functions in High-Dimensional Bayesian Optimization [5.349207553730357]
  Bayesian optimization is widely used to optimize expensive-to-evaluate black-box functions. In high-dimensional problems, finding the global maximum of the acquisition function can be difficult. We propose a better approach by employing multiple data points to leverage the historical capability of black-box optimization.
  arXiv  Detail & Related papers  (2023-02-16T13:56:32Z)
• Towards Automated Design of Bayesian Optimization via Exploratory Landscape Analysis [11.143778114800272]
  We show that a dynamic selection of the AF can benefit the BO design. We pave a way towards AutoML-assisted, on-the-fly BO designs that adjust their behavior on a run-by-run basis.
  arXiv  Detail & Related papers  (2022-11-17T17:15:04Z)
• Optimal Condition Training for Target Source Separation [56.86138859538063]
  We propose a new optimal condition training method for single-channel target source separation. We show that the complementary information carried by the diverse semantic concepts significantly helps to disentangle and isolate sources of interest.
  arXiv  Detail & Related papers  (2022-11-11T00:04:55Z)
• Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization [29.178417789839102]
  optimizing black-box functions of discrete (and potentially continuous) design parameters is a ubiquitous problem in scientific and engineering applications. We propose using probabilistic re parameterization (PR) to maximize the expectation of the acquisition function (AF) over a probability distribution. PR is complementary to (and benefits) recent work and naturally generalizes to settings with multiple objectives and black-box constraints.
  arXiv  Detail & Related papers  (2022-10-18T22:41:00Z)
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency [83.02999769628593]
  We offer a theoretical characterization of off-policy evaluation (OPE) in reinforcement learning. We show that the minimax approach enables us to achieve a fast rate of convergence for weights and quality functions. We present the first finite-sample result with first-order efficiency in non-tabular environments.
  arXiv  Detail & Related papers  (2021-02-05T03:20:39Z)
• Estimating Structural Target Functions using Machine Learning and Influence Functions [103.47897241856603]
  We propose a new framework for statistical machine learning of target functions arising as identifiable functionals from statistical models. This framework is problem- and model-agnostic and can be used to estimate a broad variety of target parameters of interest in applied statistics. We put particular focus on so-called coarsening at random/doubly robust problems with partially unobserved information.
  arXiv  Detail & Related papers  (2020-08-14T16:48:29Z)
• Causal Feature Selection for Algorithmic Fairness [61.767399505764736]
  We consider fairness in the integration component of data management. We propose an approach to identify a sub-collection of features that ensure the fairness of the dataset.
  arXiv  Detail & Related papers  (2020-06-10T20:20:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.