Diversity and Generalization in Neural Network Ensembles
- URL: http://arxiv.org/abs/2110.13786v1
- Date: Tue, 26 Oct 2021 15:41:10 GMT
- Title: Diversity and Generalization in Neural Network Ensembles
- Authors: Luis A. Ortega, Rafael Caba\~nas, Andr\'es R. Masegosa
- Abstract summary: We combine and expand previously published results in a theoretically sound framework that describes the relationship between diversity and ensemble performance.
We provide sound answers to the following questions: how to measure diversity, how diversity relates to the generalization error of an ensemble, and how diversity is promoted by neural network ensemble algorithms.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Ensembles are widely used in machine learning and, usually, provide
state-of-the-art performance in many prediction tasks. From the very beginning,
the diversity of an ensemble has been identified as a key factor for the
superior performance of these models. But the exact role that diversity plays
in ensemble models is poorly understood, specially in the context of neural
networks. In this work, we combine and expand previously published results in a
theoretically sound framework that describes the relationship between diversity
and ensemble performance for a wide range of ensemble methods. More precisely,
we provide sound answers to the following questions: how to measure diversity,
how diversity relates to the generalization error of an ensemble, and how
diversity is promoted by neural network ensemble algorithms. This analysis
covers three widely used loss functions, namely, the squared loss, the
cross-entropy loss, and the 0-1 loss; and two widely used model combination
strategies, namely, model averaging and weighted majority vote. We empirically
validate this theoretical analysis with neural network ensembles.
Related papers
- Dynamic Post-Hoc Neural Ensemblers [55.15643209328513]
In this study, we explore employing neural networks as ensemble methods.
Motivated by the risk of learning low-diversity ensembles, we propose regularizing the model by randomly dropping base model predictions.
We demonstrate this approach lower bounds the diversity within the ensemble, reducing overfitting and improving generalization capabilities.
arXiv Detail & Related papers (2024-10-06T15:25:39Z) - Generalization and Estimation Error Bounds for Model-based Neural
Networks [78.88759757988761]
We show that the generalization abilities of model-based networks for sparse recovery outperform those of regular ReLU networks.
We derive practical design rules that allow to construct model-based networks with guaranteed high generalization.
arXiv Detail & Related papers (2023-04-19T16:39:44Z) - Interpretable Diversity Analysis: Visualizing Feature Representations In
Low-Cost Ensembles [0.0]
This paper introduces several interpretability methods that can be used to qualitatively analyze diversity.
We demonstrate these techniques by comparing the diversity of feature representations between child networks using two low-cost ensemble algorithms.
arXiv Detail & Related papers (2023-02-12T00:32:03Z) - Pathologies of Predictive Diversity in Deep Ensembles [29.893614175153235]
Classic results establish that encouraging predictive diversity improves performance in ensembles of low-capacity models.
Here we demonstrate that these intuitions do not apply to high-capacity neural network ensembles (deep ensembles)
arXiv Detail & Related papers (2023-02-01T19:01:18Z) - Deep Negative Correlation Classification [82.45045814842595]
Existing deep ensemble methods naively train many different models and then aggregate their predictions.
We propose deep negative correlation classification (DNCC)
DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
arXiv Detail & Related papers (2022-12-14T07:35:20Z) - Synergies between Disentanglement and Sparsity: Generalization and
Identifiability in Multi-Task Learning [79.83792914684985]
We prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations.
Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem.
arXiv Detail & Related papers (2022-11-26T21:02:09Z) - Equivariant Transduction through Invariant Alignment [71.45263447328374]
We introduce a novel group-equivariant architecture that incorporates a group-in hard alignment mechanism.
We find that our network's structure allows it to develop stronger equivariant properties than existing group-equivariant approaches.
We additionally find that it outperforms previous group-equivariant networks empirically on the SCAN task.
arXiv Detail & Related papers (2022-09-22T11:19:45Z) - Neural Network Ensembles: Theory, Training, and the Importance of
Explicit Diversity [6.495473856599276]
Ensemble learning is a process by which multiple base learners are strategically generated and combined into one composite learner.
The right balance of learner accuracy and ensemble diversity can improve the performance of machine learning tasks on benchmark and real-world data sets.
Recent theoretical and practical work has demonstrated the subtle trade-off between accuracy and diversity in an ensemble.
arXiv Detail & Related papers (2021-09-29T00:43:57Z) - Learning distinct features helps, provably [98.78384185493624]
We study the diversity of the features learned by a two-layer neural network trained with the least squares loss.
We measure the diversity by the average $L$-distance between the hidden-layer features.
arXiv Detail & Related papers (2021-06-10T19:14:45Z) - Modeling the Evolution of Networks as Shrinking Structural Diversity [0.0]
This article reviews and evaluates models of network evolution based on the notion of structural diversity.
We show that diversity is an underlying theme of three principles of network evolution: the preferential attachment model, connectivity and link prediction.
arXiv Detail & Related papers (2020-09-21T11:30:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.