Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc
Ensemble Selection in AutoML
- URL: http://arxiv.org/abs/2307.08364v2
- Date: Wed, 2 Aug 2023 16:09:56 GMT
- Title: Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc
Ensemble Selection in AutoML
- Authors: Lennart Purucker, Lennart Schneider, Marie Anastacio, Joeran Beel,
Bernd Bischl, Holger Hoos
- Abstract summary: We introduce two novel population-based ensemble selection methods, QO-ES and QDO-ES, and compare them to greedy ensemble selection (GES)
QO-ES optimises solely for predictive performance, while QDO-ES also considers the diversity of ensembles within the population, maintaining a diverse set of well-performing ensembles during optimisation based on ideas of quality diversity optimisation.
Our results suggest that diversity can be beneficial for post hoc ensembling but also increases the risk of overfitting.
- Score: 5.089078998562186
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Automated machine learning (AutoML) systems commonly ensemble models post hoc
to improve predictive performance, typically via greedy ensemble selection
(GES). However, we believe that GES may not always be optimal, as it performs a
simple deterministic greedy search. In this work, we introduce two novel
population-based ensemble selection methods, QO-ES and QDO-ES, and compare them
to GES. While QO-ES optimises solely for predictive performance, QDO-ES also
considers the diversity of ensembles within the population, maintaining a
diverse set of well-performing ensembles during optimisation based on ideas of
quality diversity optimisation. The methods are evaluated using 71
classification datasets from the AutoML benchmark, demonstrating that QO-ES and
QDO-ES often outrank GES, albeit only statistically significant on validation
data. Our results further suggest that diversity can be beneficial for post hoc
ensembling but also increases the risk of overfitting.
Related papers
- Improving Bias Correction Standards by Quantifying its Effects on Treatment Outcomes [54.18828236350544]
Propensity score matching (PSM) addresses selection biases by selecting comparable populations for analysis.
Different matching methods can produce significantly different Average Treatment Effects (ATE) for the same task, even when meeting all validation criteria.
To address this issue, we introduce a novel metric, A2A, to reduce the number of valid matches.
arXiv Detail & Related papers (2024-07-20T12:42:24Z) - Decoding-Time Language Model Alignment with Multiple Objectives [116.42095026960598]
Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives.
Here, we propose $textbfmulti-objective decoding (MOD)$, a decoding-time algorithm that outputs the next token from a linear combination of predictions.
We show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method.
arXiv Detail & Related papers (2024-06-27T02:46:30Z) - Adaptive Preference Scaling for Reinforcement Learning with Human Feedback [103.36048042664768]
Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values.
We propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO)
Our method is versatile and can be readily adapted to various preference optimization frameworks.
arXiv Detail & Related papers (2024-06-04T20:33:22Z) - Fast Genetic Algorithm for feature selection -- A qualitative approximation approach [5.279268784803583]
We propose a two-stage surrogate-assisted evolutionary approach to address the computational issues arising from using Genetic Algorithm (GA) for feature selection.
We show that CHCQX converges faster to feature subset solutions of significantly higher accuracy, particularly for large datasets with over 100K instances.
arXiv Detail & Related papers (2024-04-05T10:15:24Z) - Class-attribute Priors: Adapting Optimization to Heterogeneity and
Fairness Objective [54.33066660817495]
Modern classification problems exhibit heterogeneities across individual classes.
We propose CAP: An effective and general method that generates a class-specific learning strategy.
We show that CAP is competitive with prior art and its flexibility unlocks clear benefits for fairness objectives beyond balanced accuracy.
arXiv Detail & Related papers (2024-01-25T17:43:39Z) - CMA-ES for Post Hoc Ensembling in AutoML: A Great Success and
Salvageable Failure [0.0]
Auto-Sklearn 1 uses only low-quality validation data for post hoc ensembling.
We compared CMA-ES, state-of-the-art gradient-free numerical optimization, to GES on the 71 classification datasets from the AutoML benchmark for AutoGluon.
For the metric balanced accuracy, CMA-ES does not overfit and outperforms GES significantly.
arXiv Detail & Related papers (2023-07-01T09:47:59Z) - DivBO: Diversity-aware CASH for Ensemble Learning [26.18421492435029]
We propose DivBO, a diversity-aware framework to inject explicit search of diversity into the CASH problems.
In the framework, we propose to use a diversity surrogate to predict the pair-wise diversity of two unseen configurations.
We show that DivBO achieves the best average ranks (1.82 and 1.73) on both validation and test errors among 10 compared methods.
arXiv Detail & Related papers (2023-02-07T04:53:21Z) - Feature Selection for Classification with QAOA [11.516147824168732]
Feature selection is of great importance in Machine Learning, where it can be used to reduce the dimensionality of classification, ranking and prediction problems.
We consider in particular a quadratic feature selection problem that can be tackled with the Approximate Quantum Algorithm Optimization (QAOA), already employed in optimization.
In our experiments, we consider seven different real-world datasets with dimensionality up to 21 and run QAOA on both a quantum simulator and, for small datasets, the 7-qubit IBM (ibm-perth) quantum computer.
arXiv Detail & Related papers (2022-11-05T09:28:53Z) - Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
Variational autoencoders (VAE) are a powerful and widely-used class of generative models.
We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs.
Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
arXiv Detail & Related papers (2021-01-06T17:36:26Z) - Automatic selection of basis-adaptive sparse polynomial chaos expansions
for engineering applications [0.0]
We describe three state-of-the-art basis-adaptive approaches for sparse chaos expansions.
We conduct an extensive benchmark in terms of global approximation accuracy on a large set of computational models.
We introduce a novel solver and basis adaptivity selection scheme guided by cross-validation error.
arXiv Detail & Related papers (2020-09-10T12:13:57Z) - Generating Diverse and Consistent QA pairs from Contexts with
Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts.
Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
arXiv Detail & Related papers (2020-05-28T08:26:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.