Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity

• URL: http://arxiv.org/abs/2109.14117v1
• Date: Wed, 29 Sep 2021 00:43:57 GMT
• Title: Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity
• Authors: Wenjing Li, Randy C. Paffenroth, David Berthiaume
• Abstract summary: Ensemble learning is a process by which multiple base learners are strategically generated and combined into one composite learner. The right balance of learner accuracy and ensemble diversity can improve the performance of machine learning tasks on benchmark and real-world data sets. Recent theoretical and practical work has demonstrated the subtle trade-off between accuracy and diversity in an ensemble.
• Score: 6.495473856599276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ensemble learning is a process by which multiple base learners are strategically generated and combined into one composite learner. There are two features that are essential to an ensemble's performance, the individual accuracies of the component learners and the overall diversity in the ensemble. The right balance of learner accuracy and ensemble diversity can improve the performance of machine learning tasks on benchmark and real-world data sets, and recent theoretical and practical work has demonstrated the subtle trade-off between accuracy and diversity in an ensemble. In this paper, we extend the extant literature by providing a deeper theoretical understanding for assessing and improving the optimality of any given ensemble, including random forests and deep neural network ensembles. We also propose a training algorithm for neural network ensembles and demonstrate that our approach provides improved performance when compared to both state-of-the-art individual learners and ensembles of state-of-the-art learners trained using standard loss functions. Our key insight is that it is better to explicitly encourage diversity in an ensemble, rather than merely allowing diversity to occur by happenstance, and that rigorous theoretical bounds on the trade-off between diversity and learner accuracy allow one to know when an optimal arrangement has been achieved.

Related papers

• On the Comparison between Multi-modal and Single-modal Contrastive Learning [50.74988548106031]
  We introduce a theoretical foundation for understanding the differences between multi-modal and single-modal contrastive learning. We identify the critical factor, which is the signal-to-noise ratio (SNR), that impacts the generalizability in downstream tasks of both multi-modal and single-modal contrastive learning. Our analysis provides a unified framework that can characterize the optimization and generalization of both single-modal and multi-modal contrastive learning.
  arXiv  Detail & Related papers  (2024-11-05T06:21:17Z)
• Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance [60.68771286221115]
  We show the interplay between sharpness and diversity within deep ensembles. We introduce SharpBalance, a training approach that balances sharpness and diversity within ensembles. Empirically, we show that SharpBalance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance.
  arXiv  Detail & Related papers  (2024-07-17T20:31:26Z)
• Diversity-Aware Agnostic Ensemble of Sharpness Minimizers [24.160975100349376]
  We propose DASH - a learning algorithm that promotes diversity and flatness within deep ensembles. We provide a theoretical backbone for our method along with extensive empirical evidence demonstrating an improvement in ensemble generalizability.
  arXiv  Detail & Related papers  (2024-03-19T23:50:11Z)
• Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learning [10.784911682565879]
  Ensembling a neural network is a widely recognized approach to enhance model performance, estimate uncertainty, and improve robustness in deep supervised learning. We present a novel self-supervised training regime that leverages an ensemble of independent sub-networks. Our method efficiently builds a sub-model ensemble with high diversity, leading to well-calibrated estimates of model uncertainty.
  arXiv  Detail & Related papers  (2023-08-28T16:58:44Z)
• Interpretable Diversity Analysis: Visualizing Feature Representations In Low-Cost Ensembles [0.0]
  This paper introduces several interpretability methods that can be used to qualitatively analyze diversity. We demonstrate these techniques by comparing the diversity of feature representations between child networks using two low-cost ensemble algorithms.
  arXiv  Detail & Related papers  (2023-02-12T00:32:03Z)
• Joint Training of Deep Ensembles Fails Due to Learner Collusion [61.557412796012535]
  Ensembles of machine learning models have been well established as a powerful method of improving performance over a single model. Traditionally, ensembling algorithms train their base learners independently or sequentially with the goal of optimizing their joint performance. We show that directly minimizing the loss of the ensemble appears to rarely be applied in practice.
  arXiv  Detail & Related papers  (2023-01-26T18:58:07Z)
• Deep Negative Correlation Classification [82.45045814842595]
  Existing deep ensemble methods naively train many different models and then aggregate their predictions. We propose deep negative correlation classification (DNCC) DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
  arXiv  Detail & Related papers  (2022-12-14T07:35:20Z)
• Efficient Diversity-Driven Ensemble for Deep Neural Networks [28.070540722925152]
  We propose Efficient Diversity-Driven Ensemble (EDDE) to address both the diversity and the efficiency of an ensemble. Compared with other well-known ensemble methods, EDDE can get highest ensemble accuracy with the lowest training cost. We evaluate EDDE on Computer Vision (CV) and Natural Language Processing (NLP) tasks.
  arXiv  Detail & Related papers  (2021-12-26T04:28:47Z)
• MultiFair: Multi-Group Fairness in Machine Learning [52.24956510371455]
  We study multi-group fairness in machine learning (MultiFair) We propose a generic end-to-end algorithmic framework to solve it. Our proposed framework is generalizable to many different settings.
  arXiv  Detail & Related papers  (2021-05-24T02:30:22Z)
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning [93.18238573921629]
  We study how Ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model. We show that ensemble/knowledge distillation in deep learning works very differently from traditional learning theory. We prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.
  arXiv  Detail & Related papers  (2020-12-17T18:34:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.