Neural Architecture for Online Ensemble Continual Learning

• URL: http://arxiv.org/abs/2211.14963v2
• Date: Mon, 21 Aug 2023 11:21:13 GMT
• Title: Neural Architecture for Online Ensemble Continual Learning
• Authors: Mateusz W\'ojcik, Witold Ko\'sciukiewicz, Tomasz Kajdanowicz, Adam Gonczarek
• Abstract summary: We present a fully differentiable ensemble method that allows us to efficiently train an ensemble of neural networks in the end-to-end regime. The proposed technique achieves SOTA results without a memory buffer and clearly outperforms the reference methods.
• Score: 6.241435193861262
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning with an increasing number of classes is a challenging task. The difficulty rises when each example is presented exactly once, which requires the model to learn online. Recent methods with classic parameter optimization procedures have been shown to struggle in such setups or have limitations like non-differentiable components or memory buffers. For this reason, we present the fully differentiable ensemble method that allows us to efficiently train an ensemble of neural networks in the end-to-end regime. The proposed technique achieves SOTA results without a memory buffer and clearly outperforms the reference methods. The conducted experiments have also shown a significant increase in the performance for small ensembles, which demonstrates the capability of obtaining relatively high classification accuracy with a reduced number of classifiers.

Related papers

• SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-training [68.7896349660824]
  We present an in-depth analysis of the progressive overfitting problem from the lens of Seq FT. Considering that the overly fast representation learning and the biased classification layer constitute this particular problem, we introduce the advanced Slow Learner with Alignment (S++) framework. Our approach involves a Slow Learner to selectively reduce the learning rate of backbone parameters, and a Alignment to align the disjoint classification layers in a post-hoc fashion.
  arXiv  Detail & Related papers  (2024-08-15T17:50:07Z)
• Contrastive-Adversarial and Diffusion: Exploring pre-training and fine-tuning strategies for sulcal identification [3.0398616939692777]
  Techniques like adversarial learning, contrastive learning, diffusion denoising learning, and ordinary reconstruction learning have become standard. The study aims to elucidate the advantages of pre-training techniques and fine-tuning strategies to enhance the learning process of neural networks.
  arXiv  Detail & Related papers  (2024-05-29T15:44:51Z)
• Domain-Agnostic Neural Architecture for Class Incremental Continual Learning in Document Processing Platform [3.630365560970225]
  Recent methods with learning gradient have been shown to struggle in such setups or have limitations like memory buffers. We present a fully differentiable architecture based on the Mixture of Experts model, that enables the training of high-performance classifiers when examples from each class are presented separately. We conducted exhaustive experiments that proved its applicability in various domains and ability to learn online in production environments.
  arXiv  Detail & Related papers  (2023-07-11T16:01:44Z)
• Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning [40.13416912075668]
  We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
  arXiv  Detail & Related papers  (2023-06-21T01:43:25Z)
• Deep Negative Correlation Classification [82.45045814842595]
  Existing deep ensemble methods naively train many different models and then aggregate their predictions. We propose deep negative correlation classification (DNCC) DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
  arXiv  Detail & Related papers  (2022-12-14T07:35:20Z)
• DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning [29.80680408934347]
  We propose an alternative framework to incremental learning where we continually fine-tune the model from a pre-trained representation. Our method takes advantage of linearization technique of a pre-trained neural network for simple and effective continual learning. We show that our method can be applied to general continual learning settings, we evaluate our method in data-incremental, task-incremental, and class-incremental learning problems.
  arXiv  Detail & Related papers  (2022-08-17T06:58:14Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Improving Calibration for Long-Tailed Recognition [68.32848696795519]
  We propose two methods to improve calibration and performance in such scenarios. For dataset bias due to different samplers, we propose shifted batch normalization. Our proposed methods set new records on multiple popular long-tailed recognition benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-01T13:55:21Z)
• Efficient Feature Transformations for Discriminative and Generative Continual Learning [98.10425163678082]
  We propose a simple task-specific feature map transformation strategy for continual learning. Theses provide powerful flexibility for learning new tasks, achieved with minimal parameters added to the base architecture. We demonstrate the efficacy and efficiency of our method with an extensive set of experiments in discriminative (CIFAR-100 and ImageNet-1K) and generative sequences of tasks.
  arXiv  Detail & Related papers  (2021-03-25T01:48:14Z)
• Incremental Embedding Learning via Zero-Shot Translation [65.94349068508863]
  Current state-of-the-art incremental learning methods tackle catastrophic forgetting problem in traditional classification networks. We propose a novel class-incremental method for embedding network, named as zero-shot translation class-incremental method (ZSTCI) In addition, ZSTCI can easily be combined with existing regularization-based incremental learning methods to further improve performance of embedding networks.
  arXiv  Detail & Related papers  (2020-12-31T08:21:37Z)
• SpaceNet: Make Free Space For Continual Learning [15.914199054779438]
  We propose a novel architectural-based method referred as SpaceNet for class incremental learning scenario. SpaceNet trains sparse deep neural networks from scratch in an adaptive way that compresses the sparse connections of each task in a compact number of neurons. Experimental results show the robustness of our proposed method against catastrophic forgetting old tasks and the efficiency of SpaceNet in utilizing the available capacity of the model.
  arXiv  Detail & Related papers  (2020-07-15T11:21:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.