Efficient Diversity-Driven Ensemble for Deep Neural Networks

• URL: http://arxiv.org/abs/2112.13316v1
• Date: Sun, 26 Dec 2021 04:28:47 GMT
• Title: Efficient Diversity-Driven Ensemble for Deep Neural Networks
• Authors: Wentao Zhang, Jiawei Jiang, Yingxia Shao, Bin Cui
• Abstract summary: We propose Efficient Diversity-Driven Ensemble (EDDE) to address both the diversity and the efficiency of an ensemble. Compared with other well-known ensemble methods, EDDE can get highest ensemble accuracy with the lowest training cost. We evaluate EDDE on Computer Vision (CV) and Natural Language Processing (NLP) tasks.
• Score: 28.070540722925152
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The ensemble of deep neural networks has been shown, both theoretically and empirically, to improve generalization accuracy on the unseen test set. However, the high training cost hinders its efficiency since we need a sufficient number of base models and each one in the ensemble has to be separately trained. Lots of methods are proposed to tackle this problem, and most of them are based on the feature that a pre-trained network can transfer its knowledge to the next base model and then accelerate the training process. However, these methods suffer a severe problem that all of them transfer knowledge without selection and thus lead to low diversity. As the effect of ensemble learning is more pronounced if ensemble members are accurate and diverse, we propose a method named Efficient Diversity-Driven Ensemble (EDDE) to address both the diversity and the efficiency of an ensemble. To accelerate the training process, we propose a novel knowledge transfer method which can selectively transfer the previous generic knowledge. To enhance diversity, we first propose a new diversity measure, then use it to define a diversity-driven loss function for optimization. At last, we adopt a Boosting-based framework to combine the above operations, such a method can also further improve diversity. We evaluate EDDE on Computer Vision (CV) and Natural Language Processing (NLP) tasks. Compared with other well-known ensemble methods, EDDE can get highest ensemble accuracy with the lowest training cost, which means it is efficient in the ensemble of neural networks.

Related papers

• Deep Boosting Learning: A Brand-new Cooperative Approach for Image-Text Matching [53.05954114863596]
  We propose a brand-new Deep Boosting Learning (DBL) algorithm for image-text matching. An anchor branch is first trained to provide insights into the data properties. A target branch is concurrently tasked with more adaptive margin constraints to further enlarge the relative distance between matched and unmatched samples.
  arXiv  Detail & Related papers  (2024-04-28T08:44:28Z)
• Scalarization for Multi-Task and Multi-Domain Learning at Scale [15.545810422759295]
  Training a single model on multiple input domains and/or output tasks allows for compressing information from multiple sources into a unified backbone. However, optimizing such networks is a challenge due to discrepancies between the different tasks or domains.
  arXiv  Detail & Related papers  (2023-10-13T07:31:04Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Multi-Task Model Personalization for Federated Supervised SVM in Heterogeneous Networks [10.169907307499916]
  Federated systems enable collaborative training on highly heterogeneous data through model personalization. To accelerate the learning procedure for diverse participants in a multi-task federated setting, more efficient and robust methods need to be developed. In this paper, we design an efficient iterative distributed method based on the alternating direction method of multipliers (ADMM) for support vector machines (SVMs) The proposed method utilizes efficient computations and model exchange in a network of heterogeneous nodes and allows personalization of the learning model in the presence of non-i.i.d. data.
  arXiv  Detail & Related papers  (2023-03-17T21:36:01Z)
• HCE: Improving Performance and Efficiency with Heterogeneously Compressed Neural Network Ensemble [22.065904428696353]
  Recent ensemble training method explores different training algorithms or settings on multiple sub-models with the same model architecture. We propose Heterogeneously Compressed Ensemble (HCE), where we build an efficient ensemble with the pruned and quantized variants from a pretrained DNN model.
  arXiv  Detail & Related papers  (2023-01-18T21:47:05Z)
• Deep Negative Correlation Classification [82.45045814842595]
  Existing deep ensemble methods naively train many different models and then aggregate their predictions. We propose deep negative correlation classification (DNCC) DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
  arXiv  Detail & Related papers  (2022-12-14T07:35:20Z)
• Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery [12.586875201983778]
  Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for training neural policies to solve complex control tasks. We show that Quality Diversity (QD) methods are a competitive alternative to information-theory-augmented RL for skill discovery.
  arXiv  Detail & Related papers  (2022-10-06T11:06:39Z)
• Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity [6.495473856599276]
  Ensemble learning is a process by which multiple base learners are strategically generated and combined into one composite learner. The right balance of learner accuracy and ensemble diversity can improve the performance of machine learning tasks on benchmark and real-world data sets. Recent theoretical and practical work has demonstrated the subtle trade-off between accuracy and diversity in an ensemble.
  arXiv  Detail & Related papers  (2021-09-29T00:43:57Z)
• Learning Neural Network Subspaces [74.44457651546728]
  Recent observations have advanced our understanding of the neural network optimization landscape. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks.
  arXiv  Detail & Related papers  (2021-02-20T23:26:58Z)
• Adaptive Serverless Learning [114.36410688552579]
  We propose a novel adaptive decentralized training approach, which can compute the learning rate from data dynamically. Our theoretical results reveal that the proposed algorithm can achieve linear speedup with respect to the number of workers. To reduce the communication-efficient overhead, we further propose a communication-efficient adaptive decentralized training approach.
  arXiv  Detail & Related papers  (2020-08-24T13:23:02Z)
• Uniform Priors for Data-Efficient Transfer [65.086680950871]
  We show that features that are most transferable have high uniformity in the embedding space. We evaluate the regularization on its ability to facilitate adaptation to unseen tasks and data.
  arXiv  Detail & Related papers  (2020-06-30T04:39:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.