Balanced Softmax Cross-Entropy for Incremental Learning

• URL: http://arxiv.org/abs/2103.12532v1
• Date: Tue, 23 Mar 2021 13:30:26 GMT
• Title: Balanced Softmax Cross-Entropy for Incremental Learning
• Authors: Quentin Jodelet, Xin Liu and Tsuyoshi Murata
• Abstract summary: Deep neural networks are prone to catastrophic forgetting when incrementally trained on new classes or new tasks. Recent methods has proven to be effective to mitigate catastrophic forgetting. We propose the use of the Balanced Softmax Cross-Entropy loss and show that it can be combined with exiting methods for incremental learning to improve their performances.
• Score: 6.5423218639215275
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks are prone to catastrophic forgetting when incrementally trained on new classes or new tasks as adaptation to the new data leads to a drastic decrease of the performance on the old classes and tasks. By using a small memory for rehearsal and knowledge distillation, recent methods has proven to be effective to mitigate catastrophic forgetting. However due to the limited size of the memory, large imbalance between the amount of data available for the old and new classes still remains which results in a deterioration of the overall accuracy of the model. To address this problem, we propose the use of the Balanced Softmax Cross-Entropy loss and show that it can be combined with exiting methods for incremental learning to improve their performances while also decreasing the computational cost of the training procedure in some cases. Complete experiments on the competitive ImageNet, subImageNet and CIFAR100 datasets show states-of-the-art results.

Related papers

• An Efficient Replay for Class-Incremental Learning with Pre-trained Models [0.0]
  In class-incremental learning, the steady state among the weight guided by each class center is disrupted, which is significantly correlated with forgetting. We propose a new method to overcoming forgetting.
  arXiv  Detail & Related papers  (2024-08-15T11:26:28Z)
• Reduced Jeffries-Matusita distance: A Novel Loss Function to Improve Generalization Performance of Deep Classification Models [0.0]
  We introduce a distance called Reduced Jeffries-Matusita as a loss function for training deep classification models to reduce the over-fitting issue. The results show that the new distance measure stabilizes the training process significantly, enhances the generalization ability, and improves the performance of the models in the Accuracy and F1-score metrics.
  arXiv  Detail & Related papers  (2024-03-13T10:51:38Z)
• Gradient Reweighting: Towards Imbalanced Class-Incremental Learning [8.438092346233054]
  Class-Incremental Learning (CIL) trains a model to continually recognize new classes from non-stationary data. A major challenge of CIL arises when applying to real-world data characterized by non-uniform distribution. We show that this dual imbalance issue causes skewed gradient updates with biased weights in FC layers, thus inducing over/under-fitting and catastrophic forgetting in CIL.
  arXiv  Detail & Related papers  (2024-02-28T18:08:03Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [100.7407460674153]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• Contextual Squeeze-and-Excitation for Efficient Few-Shot Image Classification [57.36281142038042]
  We present a new adaptive block called Contextual Squeeze-and-Excitation (CaSE) that adjusts a pretrained neural network on a new task to significantly improve performance. We also present a new training protocol based on Coordinate-Descent called UpperCaSE that exploits meta-trained CaSE blocks and fine-tuning routines for efficient adaptation.
  arXiv  Detail & Related papers  (2022-06-20T15:25:08Z)
• Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations [51.552870594221865]
  We show that last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks. We also show that last layer retraining on large ImageNet-trained models can significantly reduce reliance on background and texture information.
  arXiv  Detail & Related papers  (2022-04-06T16:55:41Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• On the Exploration of Incremental Learning for Fine-grained Image Retrieval [45.48333682748607]
  We consider the problem of fine-grained image retrieval in an incremental setting, when new categories are added over time. We propose an incremental learning method to mitigate retrieval performance degradation caused by the forgetting issue. Our method effectively mitigates the catastrophic forgetting on the original classes while achieving high performance on the new classes.
  arXiv  Detail & Related papers  (2020-10-15T21:07:44Z)
• Step-Ahead Error Feedback for Distributed Training with Compressed Gradient [99.42912552638168]
  We show that a new "gradient mismatch" problem is raised by the local error feedback in centralized distributed training. We propose two novel techniques, 1) step ahead and 2) error averaging, with rigorous theoretical analysis.
  arXiv  Detail & Related papers  (2020-08-13T11:21:07Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.