Low-Cost Self-Ensembles Based on Multi-Branch Transformation and Grouped Convolution

• URL: http://arxiv.org/abs/2408.02307v1
• Date: Mon, 5 Aug 2024 08:36:13 GMT
• Title: Low-Cost Self-Ensembles Based on Multi-Branch Transformation and Grouped Convolution
• Authors: Hojung Lee, Jong-Seok Lee,
• Abstract summary: We propose a new low-cost ensemble learning to achieve high efficiency and classification performance. For training, we employ knowledge distillation using the ensemble of the outputs as the teacher signal. Experimental results show that our method achieves state-of-the-art classification accuracy and higher uncertainty estimation performance.
• Score: 20.103367702014474
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in low-cost ensemble learning have demonstrated improved efficiency for image classification. However, the existing low-cost ensemble methods show relatively lower accuracy compared to conventional ensemble learning. In this paper, we propose a new low-cost ensemble learning, which can simultaneously achieve high efficiency and classification performance. A CNN is transformed into a multi-branch structure without introduction of additional components, which maintains the computational complexity as that of the original single model and also enhances diversity among the branches' outputs via sufficient separation between different pathways of the branches. In addition, we propose a new strategy that applies grouped convolution in the branches with different numbers of groups in different branches, which boosts the diversity of the branches' outputs. For training, we employ knowledge distillation using the ensemble of the outputs as the teacher signal. The high diversity among the outputs enables to form a powerful teacher, enhancing the individual branch's classification performance and consequently the overall ensemble performance. Experimental results show that our method achieves state-of-the-art classification accuracy and higher uncertainty estimation performance compared to previous low-cost ensemble methods. The code is available at https://github.com/hjdw2/SEMBG.

Related papers

• Achieving More with Less: A Tensor-Optimization-Powered Ensemble Method [53.170053108447455]
  Ensemble learning is a method that leverages weak learners to produce a strong learner. We design a smooth and convex objective function that leverages the concept of margin, making the strong learner more discriminative. We then compare our algorithm with random forests of ten times the size and other classical methods across numerous datasets.
  arXiv  Detail & Related papers  (2024-08-06T03:42:38Z)
• Convolutional autoencoder-based multimodal one-class classification [80.52334952912808]
  One-class classification refers to approaches of learning using data from a single class only. We propose a deep learning one-class classification method suitable for multimodal data.
  arXiv  Detail & Related papers  (2023-09-25T12:31:18Z)
• Dynamic Perceiver for Efficient Visual Recognition [87.08210214417309]
  We propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task. A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks. Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features.
  arXiv  Detail & Related papers  (2023-06-20T03:00:22Z)
• Few-shot Classification via Ensemble Learning with Multi-Order Statistics [9.145742362513932]
  We show that leveraging ensemble learning on the base classes can correspondingly reduce the true error in the novel classes. A novel method named Ensemble Learning with Multi-Order Statistics (ELMOS) is proposed in this paper. We show that our method can produce a state-of-the-art performance on multiple few-shot classification benchmark datasets.
  arXiv  Detail & Related papers  (2023-04-30T11:41:01Z)
• Deep Negative Correlation Classification [82.45045814842595]
  Existing deep ensemble methods naively train many different models and then aggregate their predictions. We propose deep negative correlation classification (DNCC) DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
  arXiv  Detail & Related papers  (2022-12-14T07:35:20Z)
• Bi-directional Feature Reconstruction Network for Fine-Grained Few-Shot Image Classification [61.411869453639845]
  We introduce a bi-reconstruction mechanism that can simultaneously accommodate for inter-class and intra-class variations. This design effectively helps the model to explore more subtle and discriminative features. Experimental results on three widely used fine-grained image classification datasets consistently show considerable improvements.
  arXiv  Detail & Related papers  (2022-11-30T16:55:14Z)
• Structurally Diverse Sampling Reduces Spurious Correlations in Semantic Parsing Datasets [51.095144091781734]
  We propose a novel algorithm for sampling a structurally diverse set of instances from a labeled instance pool with structured outputs. We show that our algorithm performs competitively with or better than prior algorithms in not only compositional template splits but also traditional IID splits. In general, we find that diverse train sets lead to better generalization than random training sets of the same size in 9 out of 10 dataset-split pairs.
  arXiv  Detail & Related papers  (2022-03-16T07:41:27Z)
• On-the-Fly Ensemble Pruning in Evolving Data Streams [4.137914981603379]
  CCRP is an on-the-fly ensemble prun-ing method for multi-class data stream classification. We show that different types of en-sembles that integrate CCRP consistently yield on par or superior performance with 20% to 90% less averagememory consumption.
  arXiv  Detail & Related papers  (2021-09-15T22:54:22Z)
• Multiple Classifiers Based Maximum Classifier Discrepancy for Unsupervised Domain Adaptation [25.114533037440896]
  We propose to extend the structure of two classifiers to multiple classifiers to further boost its performance. We demonstrate that, on average, adopting the structure of three classifiers normally yields the best performance as a trade-off between the accuracy and efficiency.
  arXiv  Detail & Related papers  (2021-08-02T03:00:13Z)
• Residual Likelihood Forests [19.97069303172077]
  This paper presents a novel ensemble learning approach called Residual Likelihood Forests (RLF) Our weak learners produce conditional likelihoods that are sequentially optimized using global loss in the context of previous learners. When compared against several ensemble approaches including Random Forests and Gradient Boosted Trees, RLFs offer a significant improvement in performance.
  arXiv  Detail & Related papers  (2020-11-04T00:59:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.