Differentiable Model Selection for Ensemble Learning
- URL: http://arxiv.org/abs/2211.00251v2
- Date: Fri, 19 May 2023 16:59:33 GMT
- Title: Differentiable Model Selection for Ensemble Learning
- Authors: James Kotary, Vincenzo Di Vito, Ferdinando Fioretto
- Abstract summary: This paper proposes a novel framework for differentiable model selection integrating machine learning and optimization.
The framework is tailored for ensemble learning, a strategy that combines the outputs of individually pre-trained models, and learns to select appropriate ensemble members for a particular input sample.
- Score: 37.99501959301896
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Model selection is a strategy aimed at creating accurate and robust models. A
key challenge in designing these algorithms is identifying the optimal model
for classifying any particular input sample. This paper addresses this
challenge and proposes a novel framework for differentiable model selection
integrating machine learning and combinatorial optimization. The framework is
tailored for ensemble learning, a strategy that combines the outputs of
individually pre-trained models, and learns to select appropriate ensemble
members for a particular input sample by transforming the ensemble learning
task into a differentiable selection program trained end-to-end within the
ensemble learning model. Tested on various tasks, the proposed framework
demonstrates its versatility and effectiveness, outperforming conventional and
advanced consensus rules across a variety of settings and learning tasks.
Related papers
- Hit the Sweet Spot! Span-Level Ensemble for Large Language Models [8.34562564266839]
We propose SweetSpan, a span-level ensemble method that effectively balances the need for real-time adjustments and the information required for accurate ensemble decisions.
Our approach involves two key steps: First, we have each candidate model independently generate candidate spans based on the shared prefix.
Second, we calculate perplexity scores to facilitate mutual evaluation among the candidate models and achieve robust span selection by filtering out unfaithful scores.
arXiv Detail & Related papers (2024-09-27T09:41:29Z) - An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting [53.36437745983783]
We first construct a max-margin optimization-based model to model potentially non-monotonic preferences.
We devise information amount measurement methods and question selection strategies to pinpoint the most informative alternative in each iteration.
Two incremental preference elicitation-based algorithms are developed to learn potentially non-monotonic preferences.
arXiv Detail & Related papers (2024-09-04T14:36:20Z) - A tutorial on learning from preferences and choices with Gaussian Processes [0.7234862895932991]
This tutorial builds upon established research while introducing some novel GP-based models to address specific gaps in the existing literature.
This framework enables the construction of preference learning models that encompass random utility models, limits of discernment, and scenarios with multiple conflicting utilities for both object- and label-preference.
arXiv Detail & Related papers (2024-03-18T13:40:48Z) - Universal Semi-supervised Model Adaptation via Collaborative Consistency
Training [92.52892510093037]
We introduce a realistic and challenging domain adaptation problem called Universal Semi-supervised Model Adaptation (USMA)
We propose a collaborative consistency training framework that regularizes the prediction consistency between two models.
Experimental results demonstrate the effectiveness of our method on several benchmark datasets.
arXiv Detail & Related papers (2023-07-07T08:19:40Z) - Data-Free Diversity-Based Ensemble Selection For One-Shot Federated
Learning in Machine Learning Model Market [2.9046424358155236]
We present a novel Data-Free Diversity-Based method called DeDES to address the ensemble selection problem for models generated by one-shot federated learning.
Our method can achieve both better performance and higher efficiency over 5 datasets and 4 different model structures.
arXiv Detail & Related papers (2023-02-23T02:36:27Z) - Autoselection of the Ensemble of Convolutional Neural Networks with
Second-Order Cone Programming [0.8029049649310213]
This study proposes a mathematical model which prunes the ensemble of Convolutional Neural Networks (CNN)
The proposed model is tested on CIFAR-10, CIFAR-100 and MNIST data sets.
arXiv Detail & Related papers (2023-02-12T16:18:06Z) - Model ensemble instead of prompt fusion: a sample-specific knowledge
transfer method for few-shot prompt tuning [85.55727213502402]
We focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks.
We propose Sample-specific Ensemble of Source Models (SESoM)
SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs.
arXiv Detail & Related papers (2022-10-23T01:33:16Z) - Auxiliary Class Based Multiple Choice Learning [0.0]
We propose an advanced ensemble method, called Auxiliary class based Multiple Choice Learning (AMCL), to ultimately specialize each model under the framework of multiple choice learning (MCL)
The performance of AMCL exceeds all others in most of the public datasets trained with various networks as members of the ensembles.
arXiv Detail & Related papers (2021-08-06T05:55:46Z) - Learning to Select Base Classes for Few-shot Classification [96.92372639495551]
We use the Similarity Ratio as an indicator for the generalization performance of a few-shot model.
We then formulate the base class selection problem as a submodular optimization problem over Similarity Ratio.
arXiv Detail & Related papers (2020-04-01T09:55:18Z) - Stepwise Model Selection for Sequence Prediction via Deep Kernel
Learning [100.83444258562263]
We propose a novel Bayesian optimization (BO) algorithm to tackle the challenge of model selection in this setting.
In order to solve the resulting multiple black-box function optimization problem jointly and efficiently, we exploit potential correlations among black-box functions.
We are the first to formulate the problem of stepwise model selection (SMS) for sequence prediction, and to design and demonstrate an efficient joint-learning algorithm for this purpose.
arXiv Detail & Related papers (2020-01-12T09:42:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.