The Forward-Forward Algorithm: Some Preliminary Investigations
- URL: http://arxiv.org/abs/2212.13345v1
- Date: Tue, 27 Dec 2022 02:54:46 GMT
- Title: The Forward-Forward Algorithm: Some Preliminary Investigations
- Authors: Geoffrey Hinton
- Abstract summary: The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes.
If the positive and negative passes could be separated in time, the negative passes could be done offline.
- Score: 91.3755431537592
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The aim of this paper is to introduce a new learning procedure for neural
networks and to demonstrate that it works well enough on a few small problems
to be worth further investigation. The Forward-Forward algorithm replaces the
forward and backward passes of backpropagation by two forward passes, one with
positive (i.e. real) data and the other with negative data which could be
generated by the network itself. Each layer has its own objective function
which is simply to have high goodness for positive data and low goodness for
negative data. The sum of the squared activities in a layer can be used as the
goodness but there are many other possibilities, including minus the sum of the
squared activities. If the positive and negative passes could be separated in
time, the negative passes could be done offline, which would make the learning
much simpler in the positive pass and allow video to be pipelined through the
network without ever storing activities or stopping to propagate derivatives.
Related papers
- Forward Learning of Graph Neural Networks [17.79590285482424]
Backpropagation (BP) is the de facto standard for training deep neural networks (NNs)
BP imposes several constraints, which are not only biologically implausible, but also limit the scalability, parallelism, and flexibility in learning NNs.
We propose ForwardGNN, which avoids the constraints imposed by BP via an effective layer-wise local forward training.
arXiv Detail & Related papers (2024-03-16T19:40:35Z) - Rethinking Negative Pairs in Code Search [56.23857828689406]
We propose a simple yet effective Soft-InfoNCE loss that inserts weight terms into InfoNCE.
We analyze the effects of Soft-InfoNCE on controlling the distribution of learnt code representations and on deducing a more precise mutual information estimation.
arXiv Detail & Related papers (2023-10-12T06:32:42Z) - Using Linear Regression for Iteratively Training Neural Networks [4.873362301533824]
We present a simple linear regression based approach for learning the weights and biases of a neural network.
The approach is intended to be to larger, more complex architectures.
arXiv Detail & Related papers (2023-07-11T11:53:25Z) - Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls
and New Benchmarking [66.83273589348758]
Link prediction attempts to predict whether an unseen edge exists based on only a portion of edges of a graph.
A flurry of methods have been introduced in recent years that attempt to make use of graph neural networks (GNNs) for this task.
New and diverse datasets have also been created to better evaluate the effectiveness of these new models.
arXiv Detail & Related papers (2023-06-18T01:58:59Z) - SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval [126.22182758461244]
We show that according to the measured relevance scores, the negatives ranked around the positives are generally more informative and less likely to be false negatives.
We propose a simple ambiguous negatives sampling method, SimANS, which incorporates a new sampling probability distribution to sample more ambiguous negatives.
arXiv Detail & Related papers (2022-10-21T07:18:05Z) - Generating Negative Samples for Sequential Recommendation [83.60655196391855]
We propose to Generate Negative Samples (items) for Sequential Recommendation (SR)
A negative item is sampled at each time step based on the current SR model's learned user preferences toward items.
Experiments on four public datasets verify the importance of providing high-quality negative samples for SR.
arXiv Detail & Related papers (2022-08-07T05:44:13Z) - Recursive Least-Squares Estimator-Aided Online Learning for Visual
Tracking [58.14267480293575]
We propose a simple yet effective online learning approach for few-shot online adaptation without requiring offline training.
It allows an in-built memory retention mechanism for the model to remember the knowledge about the object seen before.
We evaluate our approach based on two networks in the online learning families for tracking, i.e., multi-layer perceptrons in RT-MDNet and convolutional neural networks in DiMP.
arXiv Detail & Related papers (2021-12-28T06:51:18Z) - Revisiting the Negative Data of Distantly Supervised Relation Extraction [17.00557139562208]
Distantly supervision automatically generates plenty of training samples for relation extraction.
It also incurs two major problems: noisy labels and imbalanced training data.
We propose a pipeline approach, dubbed textscReRe, that performs sentence-level relation detection then subject/object extraction.
arXiv Detail & Related papers (2021-05-21T06:44:19Z) - Training Neural Networks Using the Property of Negative Feedback to
Inverse a Function [0.0]
This paper describes how the property of a negative feedback system to perform inverse of a function can be used for training neural networks.
We have applied this method to the MNIST dataset and obtained results that shows the method is viable for neural network training.
arXiv Detail & Related papers (2021-03-25T20:13:53Z) - SCE: Scalable Network Embedding from Sparsest Cut [20.08464038805681]
Large-scale network embedding is to learn a latent representation for each node in an unsupervised manner.
A key of success to such contrastive learning methods is how to draw positive and negative samples.
In this paper, we propose SCE for unsupervised network embedding only using negative samples for training.
arXiv Detail & Related papers (2020-06-30T03:18:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.