CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion

• URL: http://arxiv.org/abs/2405.09637v2
• Date: Sat, 8 Jun 2024 21:02:15 GMT
• Title: CLASSP: a Biologically-Inspired Approach to Continual Learning through Adjustment Suppression and Sparsity Promotion
• Authors: Oswaldo Ludwig,
• Abstract summary: This paper introduces a new training method named Continual Learning through Adjustment Suppression and Sparsity Promotion (CLASSP) CLASSP is based on two main principles observed in neuroscience, particularly in the context of synaptic transmission and Long-Term Potentiation. When compared with Elastic Weight Consolidation (EWC) datasets, CLASSP demonstrates superior performance in terms of accuracy and memory footprint.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces a new biologically-inspired training method named Continual Learning through Adjustment Suppression and Sparsity Promotion (CLASSP). CLASSP is based on two main principles observed in neuroscience, particularly in the context of synaptic transmission and Long-Term Potentiation (LTP). The first principle is a decay rate over the weight adjustment, which is implemented as a generalization of the AdaGrad optimization algorithm. This means that weights that have received many updates should have lower learning rates as they likely encode important information about previously seen data. However, this principle results in a diffuse distribution of updates throughout the model, as it promotes updates for weights that haven't been previously updated, while a sparse update distribution is preferred to leave weights unassigned for future tasks. Therefore, the second principle introduces a threshold on the loss gradient. This promotes sparse learning by updating a weight only if the loss gradient with respect to that weight is above a certain threshold, i.e. only updating weights with a significant impact on the current loss. Both principles reflect phenomena observed in LTP, where a threshold effect and a gradual saturation of potentiation have been observed. CLASSP is implemented in a Python/PyTorch class, making it applicable to any model. When compared with Elastic Weight Consolidation (EWC) using Computer Vision and sentiment analysis datasets, CLASSP demonstrates superior performance in terms of accuracy and memory footprint.

Related papers

• Few-Shot Class-Incremental Learning with Prior Knowledge [94.95569068211195]
  We propose Learning with Prior Knowledge (LwPK) to enhance the generalization ability of the pre-trained model. Experimental results indicate that LwPK effectively enhances the model resilience against catastrophic forgetting.
  arXiv  Detail & Related papers  (2024-02-02T08:05:35Z)
• One Step Learning, One Step Review [6.540346282603399]
  We propose a novel weight rollback-based fine-tuning method called OLOR (One step Learning, One step Review) In this paper, we propose a novel weight rollback-based fine-tuning method called OLOR (One step Learning, One step Review)
  arXiv  Detail & Related papers  (2024-01-19T11:45:31Z)
• InRank: Incremental Low-Rank Learning [85.6380047359139]
  gradient-based training implicitly regularizes neural networks towards low-rank solutions through a gradual increase of the rank during training. Existing training algorithms do not exploit the low-rank property to improve computational efficiency. We design a new training algorithm Incremental Low-Rank Learning (InRank), which explicitly expresses cumulative weight updates as low-rank matrices.
  arXiv  Detail & Related papers  (2023-06-20T03:03:04Z)
• Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transport [78.9167477093745]
  We propose a novel distribution calibration method by learning the adaptive weight matrix between novel samples and base classes. Experimental results on standard benchmarks demonstrate that our proposed plug-and-play model outperforms competing approaches.
  arXiv  Detail & Related papers  (2022-10-09T02:32:57Z)
• Long-Tailed Recognition via Weight Balancing [66.03068252811993]
  Naive training produces models that are biased toward common classes in terms of higher accuracy. We investigate three techniques to balance weights, L2-normalization, weight decay, and MaxNorm. Our approach achieves the state-of-the-art accuracy on five standard benchmarks.
  arXiv  Detail & Related papers  (2022-03-27T03:26:31Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• FixNorm: Dissecting Weight Decay for Training Deep Neural Networks [7.820667552233989]
  We propose a new training method called FixNorm, which discards weight decay and directly controls the two mechanisms. On ImageNet classification task, training EfficientNet-B0 with FixNorm achieves 77.7%, which outperforms the original baseline by a clear margin.
  arXiv  Detail & Related papers  (2021-03-29T05:41:56Z)
• The Golden Ratio of Learning and Momentum [0.5076419064097732]
  This paper proposes a new information-theoretical loss function motivated by neural signal processing in a synapse. All results taken together show that loss, learning rate, and momentum are closely connected.
  arXiv  Detail & Related papers  (2020-06-08T17:08:13Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.