Incremental Class Learning using Variational Autoencoders with Similarity Learning

• URL: http://arxiv.org/abs/2110.01303v1
• Date: Mon, 4 Oct 2021 10:19:53 GMT
• Title: Incremental Class Learning using Variational Autoencoders with Similarity Learning
• Authors: Jiahao Huo, Terence L. van Zyl
• Abstract summary: Catastrophic forgetting in neural networks during incremental learning remains a challenging problem. Our research investigates catastrophic forgetting for four well-known metric-based loss functions during incremental class learning. The angular loss was least affected, followed by contrastive, triplet loss, and centre loss with good mining techniques.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Catastrophic forgetting in neural networks during incremental learning remains a challenging problem. Previous research investigated catastrophic forgetting in fully connected networks, with some earlier work exploring activation functions and learning algorithms. Applications of neural networks have been extended to include similarity and metric learning. It is of significant interest to understand how metric learning loss functions would be affected by catastrophic forgetting. Our research investigates catastrophic forgetting for four well-known metric-based loss functions during incremental class learning. The loss functions are angular, contrastive, centre, and triplet loss. Our results show that the rate of catastrophic forgetting is different across loss functions on multiple datasets. The angular loss was least affected, followed by contrastive, triplet loss, and centre loss with good mining techniques. We implemented three existing incremental learning techniques, iCARL, EWC, and EBLL. We further proposed our novel technique using VAEs to generate representation as exemplars that are passed through intermediate layers of the network. Our method outperformed the three existing techniques. We have shown that we do not require stored images as exemplars for incremental learning with similarity learning. The generated representations can help preserve regions of the embedding space used by prior knowledge so that new knowledge will not "overwrite" prior knowledge.

Related papers

• Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms [80.37846867546517]
  We show how to train eight different neural networks with custom objectives. We exploit their second-order information via their empirical Fisherssian matrices. We apply Loss Lossiable algorithms to achieve significant improvements for less differentiable algorithms.
  arXiv  Detail & Related papers  (2024-10-24T18:02:11Z)
• A Methodology-Oriented Study of Catastrophic Forgetting in Incremental Deep Neural Networks [4.844142799909677]
  The ability of learning throughout their lifespan is referred as continuous learning which is using neurocognition mechanism. The main challenge in artificial intelligence is that the incremental learning of the autonomous agent when new data confronted. Here we are comparing three types of incremental learning methods: (1) Exemplar based methods, (2) Memory based methods, and (3) Network based method.
  arXiv  Detail & Related papers  (2024-05-11T05:10:07Z)
• Class Anchor Margin Loss for Content-Based Image Retrieval [97.81742911657497]
  We propose a novel repeller-attractor loss that falls in the metric learning paradigm, yet directly optimize for the L2 metric without the need of generating pairs. We evaluate the proposed objective in the context of few-shot and full-set training on the CBIR task, by using both convolutional and transformer architectures.
  arXiv  Detail & Related papers  (2023-06-01T12:53:10Z)
• SuSana Distancia is all you need: Enforcing class separability in metric learning via two novel distance-based loss functions for few-shot image classification [0.9236074230806579]
  We propose two loss functions which consider the importance of the embedding vectors by looking at the intra-class and inter-class distance between the few data. Our results show a significant improvement in accuracy in the miniImagenNet benchmark compared to other metric-based few-shot learning methods by a margin of 2%.
  arXiv  Detail & Related papers  (2023-05-15T23:12:09Z)
• Online Loss Function Learning [13.744076477599707]
  Loss function learning aims to automate the task of designing a loss function for a machine learning model. We propose a new loss function learning technique for adaptively updating the loss function online after each update to the base model parameters.
  arXiv  Detail & Related papers  (2023-01-30T19:22:46Z)
• On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
  Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
  arXiv  Detail & Related papers  (2022-03-08T09:57:48Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)
• Do Lessons from Metric Learning Generalize to Image-Caption Retrieval? [67.45267657995748]
  The triplet loss with semi-hard negatives has become the de facto choice for image-caption retrieval (ICR) methods that are optimized from scratch. Recent progress in metric learning has given rise to new loss functions that outperform the triplet loss on tasks such as image retrieval and representation learning. We ask whether these findings generalize to the setting of ICR by comparing three loss functions on two ICR methods.
  arXiv  Detail & Related papers  (2022-02-14T15:18:00Z)
• Incremental Embedding Learning via Zero-Shot Translation [65.94349068508863]
  Current state-of-the-art incremental learning methods tackle catastrophic forgetting problem in traditional classification networks. We propose a novel class-incremental method for embedding network, named as zero-shot translation class-incremental method (ZSTCI) In addition, ZSTCI can easily be combined with existing regularization-based incremental learning methods to further improve performance of embedding networks.
  arXiv  Detail & Related papers  (2020-12-31T08:21:37Z)
• A Comparison of Metric Learning Loss Functions for End-To-End Speaker Verification [4.617249742207066]
  We compare several metric learning loss functions in a systematic manner on the VoxCeleb dataset. We show that the additive angular margin loss function outperforms all other loss functions in the study. Based on a combination of SincNet trainable features and the x-vector architecture, the network used in this paper brings us a step closer to a really-end-to-end speaker verification system.
  arXiv  Detail & Related papers  (2020-03-31T08:36:07Z)
• Adversarial Incremental Learning [0.0]
  Deep learning can forget previously learned information upon learning new tasks where previous data is not available. We propose an adversarial discriminator based method that does not make use of old data at all while training on new tasks. We are able to outperform other state-of-the-art methods on CIFAR-100, SVHN, and MNIST datasets.
  arXiv  Detail & Related papers  (2020-01-30T02:25:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.