Convolutional Channel-wise Competitive Learning for the Forward-Forward
Algorithm
- URL: http://arxiv.org/abs/2312.12668v1
- Date: Tue, 19 Dec 2023 23:48:43 GMT
- Title: Convolutional Channel-wise Competitive Learning for the Forward-Forward
Algorithm
- Authors: Andreas Papachristodoulou, Christos Kyrkou, Stelios Timotheou,
Theocharis Theocharides
- Abstract summary: Forward-Forward (FF) algorithm has been proposed to alleviate the issues of backpropagation (BP) commonly used to train deep neural networks.
We take the main ideas of FF and improve them by leveraging channel-wise competitive learning in the context of convolutional neural networks for image classification tasks.
Our method outperforms recent FF-based models on image classification tasks, achieving testing errors of 0.58%, 7.69%, 21.89%, and 48.77% on MNIST, Fashion-MNIST, CIFAR-10 and CIFAR-100 respectively.
- Score: 5.1246638322893245
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: The Forward-Forward (FF) Algorithm has been recently proposed to alleviate
the issues of backpropagation (BP) commonly used to train deep neural networks.
However, its current formulation exhibits limitations such as the generation of
negative data, slower convergence, and inadequate performance on complex tasks.
In this paper, we take the main ideas of FF and improve them by leveraging
channel-wise competitive learning in the context of convolutional neural
networks for image classification tasks. A layer-wise loss function is
introduced that promotes competitive learning and eliminates the need for
negative data construction. To enhance both the learning of compositional
features and feature space partitioning, a channel-wise feature separator and
extractor block is proposed that complements the competitive learning process.
Our method outperforms recent FF-based models on image classification tasks,
achieving testing errors of 0.58%, 7.69%, 21.89%, and 48.77% on MNIST,
Fashion-MNIST, CIFAR-10 and CIFAR-100 respectively. Our approach bridges the
performance gap between FF learning and BP methods, indicating the potential of
our proposed approach to learn useful representations in a layer-wise modular
fashion, enabling more efficient and flexible learning.
Related papers
- LeRF: Learning Resampling Function for Adaptive and Efficient Image Interpolation [64.34935748707673]
Recent deep neural networks (DNNs) have made impressive progress in performance by introducing learned data priors.
We propose a novel method of Learning Resampling (termed LeRF) which takes advantage of both the structural priors learned by DNNs and the locally continuous assumption.
LeRF assigns spatially varying resampling functions to input image pixels and learns to predict the shapes of these resampling functions with a neural network.
arXiv Detail & Related papers (2024-07-13T16:09:45Z) - Bayesian Learning-driven Prototypical Contrastive Loss for Class-Incremental Learning [42.14439854721613]
We propose a prototypical network with a Bayesian learning-driven contrastive loss (BLCL) tailored specifically for class-incremental learning scenarios.
Our approach dynamically adapts the balance between the cross-entropy and contrastive loss functions with a Bayesian learning technique.
arXiv Detail & Related papers (2024-05-17T19:49:02Z) - An NMF-Based Building Block for Interpretable Neural Networks With
Continual Learning [0.8158530638728501]
Existing learning methods often struggle to balance interpretability and predictive performance.
Our approach aims to strike a better balance between these two aspects through the use of a building block based on NMF.
arXiv Detail & Related papers (2023-11-20T02:00:33Z) - FaFCNN: A General Disease Classification Framework Based on Feature
Fusion Neural Networks [4.097623533226476]
We propose the Feature-aware Fusion Correlation Neural Network (FaFCNN), which introduces a feature-aware interaction module and a feature alignment module based on domain adversarial learning.
The experimental results show that training using augmented features obtained by pre-training gradient boosting decision tree yields more performance gains than random-forest based methods.
arXiv Detail & Related papers (2023-07-24T04:23:08Z) - Stochastic Unrolled Federated Learning [85.6993263983062]
We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning.
Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
arXiv Detail & Related papers (2023-05-24T17:26:22Z) - The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF.
Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples.
In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
arXiv Detail & Related papers (2023-03-17T02:01:11Z) - Improved Algorithms for Neural Active Learning [74.89097665112621]
We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting.
We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
arXiv Detail & Related papers (2022-10-02T05:03:38Z) - Improving Music Performance Assessment with Contrastive Learning [78.8942067357231]
This study investigates contrastive learning as a potential method to improve existing MPA systems.
We introduce a weighted contrastive loss suitable for regression tasks applied to a convolutional neural network.
Our results show that contrastive-based methods are able to match and exceed SoTA performance for MPA regression tasks.
arXiv Detail & Related papers (2021-08-03T19:24:25Z) - A Hybrid Method for Training Convolutional Neural Networks [3.172761915061083]
We propose a hybrid method that uses both backpropagation and evolutionary strategies to train Convolutional Neural Networks.
We show that the proposed hybrid method is capable of improving upon regular training in the task of image classification.
arXiv Detail & Related papers (2020-04-15T17:52:48Z) - Belief Propagation Reloaded: Learning BP-Layers for Labeling Problems [83.98774574197613]
We take one of the simplest inference methods, a truncated max-product Belief propagation, and add what is necessary to make it a proper component of a deep learning model.
This BP-Layer can be used as the final or an intermediate block in convolutional neural networks (CNNs)
The model is applicable to a range of dense prediction problems, is well-trainable and provides parameter-efficient and robust solutions in stereo, optical flow and semantic segmentation.
arXiv Detail & Related papers (2020-03-13T13:11:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.