A Framework for Generating Informative Benchmark Instances
- URL: http://arxiv.org/abs/2205.14753v1
- Date: Sun, 29 May 2022 19:56:08 GMT
- Title: A Framework for Generating Informative Benchmark Instances
- Authors: Nguyen Dang, \"Ozg\"ur Akg\"un, Joan Espasa, Ian Miguel, Peter
Nightingale
- Abstract summary: Benchmarking is an important tool for assessing the relative performance of alternative solving approaches.
Modern constraint programming languages allow the specification of a class-level model that is parameterised over instance data.
We introduce a framework that combines these two properties to generate a large number of benchmark instances.
- Score: 3.8848561367220276
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Benchmarking is an important tool for assessing the relative performance of
alternative solving approaches. However, the utility of benchmarking is limited
by the quantity and quality of the available problem instances. Modern
constraint programming languages typically allow the specification of a
class-level model that is parameterised over instance data. This separation
presents an opportunity for automated approaches to generate instance data that
define instances that are graded (solvable at a certain difficulty level for a
solver) or can discriminate between two solving approaches. In this paper, we
introduce a framework that combines these two properties to generate a large
number of benchmark instances, purposely generated for effective and
informative benchmarking. We use five problems that were used in the MiniZinc
competition to demonstrate the usage of our framework. In addition to producing
a ranking among solvers, our framework gives a broader understanding of the
behaviour of each solver for the whole instance space; for example by finding
subsets of instances where the solver performance significantly varies from its
average performance.
Related papers
- POGEMA: A Benchmark Platform for Cooperative Multi-Agent Navigation [76.67608003501479]
We introduce and specify an evaluation protocol defining a range of domain-related metrics computed on the basics of the primary evaluation indicators.
The results of such a comparison, which involves a variety of state-of-the-art MARL, search-based, and hybrid methods, are presented.
arXiv Detail & Related papers (2024-07-20T16:37:21Z) - Generating collective counterfactual explanations in score-based
classification via mathematical optimization [4.281723404774889]
A counterfactual explanation of an instance indicates how this instance should be minimally modified so that the perturbed instance is classified in the desired class.
Most of the Counterfactual Analysis literature focuses on the single-instance single-counterfactual setting.
By means of novel Mathematical Optimization models, we provide a counterfactual explanation for each instance in a group of interest.
arXiv Detail & Related papers (2023-10-19T15:18:42Z) - Generalization Properties of Retrieval-based Models [50.35325326050263]
Retrieval-based machine learning methods have enjoyed success on a wide range of problems.
Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored.
We present a formal treatment of retrieval-based models to characterize their generalization ability.
arXiv Detail & Related papers (2022-10-06T00:33:01Z) - Rethinking Clustering-Based Pseudo-Labeling for Unsupervised
Meta-Learning [146.11600461034746]
Method for unsupervised meta-learning, CACTUs, is a clustering-based approach with pseudo-labeling.
This approach is model-agnostic and can be combined with supervised algorithms to learn from unlabeled data.
We prove that the core reason for this is lack of a clustering-friendly property in the embedding space.
arXiv Detail & Related papers (2022-09-27T19:04:36Z) - An Additive Instance-Wise Approach to Multi-class Model Interpretation [53.87578024052922]
Interpretable machine learning offers insights into what factors drive a certain prediction of a black-box system.
Existing methods mainly focus on selecting explanatory input features, which follow either locally additive or instance-wise approaches.
This work exploits the strengths of both methods and proposes a global framework for learning local explanations simultaneously for multiple target classes.
arXiv Detail & Related papers (2022-07-07T06:50:27Z) - HyperImpute: Generalized Iterative Imputation with Automatic Model
Selection [77.86861638371926]
We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models.
We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
arXiv Detail & Related papers (2022-06-15T19:10:35Z) - The Benchmark Lottery [114.43978017484893]
"A benchmark lottery" describes the overall fragility of the machine learning benchmarking process.
We show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks.
arXiv Detail & Related papers (2021-07-14T21:08:30Z) - Transductive Few-Shot Learning: Clustering is All You Need? [31.21306826132773]
We investigate a general formulation for transive few-shot learning, which integrates prototype-based objectives.
We find that our method yields competitive performances, in term of accuracy and optimization, while scaling up to large problems.
Surprisingly, we find that our general model already achieve competitive performances in comparison to the state-of-the-art learning.
arXiv Detail & Related papers (2021-06-16T16:14:01Z) - HAWKS: Evolving Challenging Benchmark Sets for Cluster Analysis [2.5329716878122404]
Comprehensive benchmarking of clustering algorithms is difficult.
There is no consensus regarding the best practice for rigorous benchmarking.
We demonstrate the important role evolutionary algorithms play to support flexible generation of such benchmarks.
arXiv Detail & Related papers (2021-02-13T15:01:34Z) - Exploring Instance Generation for Automated Planning [1.6735240552964108]
Constraint programming and automated planning are examples of these areas.
The efficiency of each solving method varies not only between problems, but also between instances of the same problem.
We propose a new approach where the whole planning problem description is modelled using Essence.
arXiv Detail & Related papers (2020-09-21T19:58:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.