Training Feedforward Neural Networks with Bayesian Hyper-Heuristics
- URL: http://arxiv.org/abs/2303.16912v1
- Date: Wed, 29 Mar 2023 11:40:28 GMT
- Title: Training Feedforward Neural Networks with Bayesian Hyper-Heuristics
- Authors: Arn\'e Schreuder, Anna Bosman, Andries Engelbrecht, Christopher
Cleghorn
- Abstract summary: This research introduces a novel population-based Bayesian hyper-heuristic (BHH) that is used to train feedforward networks (FFNNs)
The performance of the BHH is compared to that of ten popular low-level neurals, each with different search behaviours.
The BHH is shown to be able to train FFNNs well and provide an automated method for finding the best to train the FFNNs at various stages of the training process.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The process of training feedforward neural networks (FFNNs) can benefit from
an automated process where the best heuristic to train the network is sought
out automatically by means of a high-level probabilistic-based heuristic. This
research introduces a novel population-based Bayesian hyper-heuristic (BHH)
that is used to train feedforward neural networks (FFNNs). The performance of
the BHH is compared to that of ten popular low-level heuristics, each with
different search behaviours. The chosen heuristic pool consists of classic
gradient-based heuristics as well as meta-heuristics (MHs). The empirical
process is executed on fourteen datasets consisting of classification and
regression problems with varying characteristics. The BHH is shown to be able
to train FFNNs well and provide an automated method for finding the best
heuristic to train the FFNNs at various stages of the training process.
Related papers
- Speed Limits for Deep Learning [67.69149326107103]
Recent advancement in thermodynamics allows bounding the speed at which one can go from the initial weight distribution to the final distribution of the fully trained network.
We provide analytical expressions for these speed limits for linear and linearizable neural networks.
Remarkably, given some plausible scaling assumptions on the NTK spectra and spectral decomposition of the labels -- learning is optimal in a scaling sense.
arXiv Detail & Related papers (2023-07-27T06:59:46Z) - How neural networks learn to classify chaotic time series [77.34726150561087]
We study the inner workings of neural networks trained to classify regular-versus-chaotic time series.
We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
arXiv Detail & Related papers (2023-06-04T08:53:27Z) - SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural
Networks [56.35403810762512]
Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware.
We study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method.
arXiv Detail & Related papers (2023-02-01T04:22:59Z) - Online Training Through Time for Spiking Neural Networks [66.7744060103562]
Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models.
Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency.
We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
arXiv Detail & Related papers (2022-10-09T07:47:56Z) - Neural Capacitance: A New Perspective of Neural Network Selection via
Edge Dynamics [85.31710759801705]
Current practice requires expensive computational costs in model training for performance prediction.
We propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training.
Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections.
arXiv Detail & Related papers (2022-01-11T20:53:15Z) - Can we learn gradients by Hamiltonian Neural Networks? [68.8204255655161]
We propose a meta-learner based on ODE neural networks that learns gradients.
We demonstrate that our method outperforms a meta-learner based on LSTM for an artificial task and the MNIST dataset with ReLU activations in the optimizee.
arXiv Detail & Related papers (2021-10-31T18:35:10Z) - Path classification by stochastic linear recurrent neural networks [2.5499055723658097]
We show that RNNs retain a partial signature of the paths they are fed as the unique information exploited for training and classification tasks.
We argue that these RNNs are easy to train and robust and back these observations with numerical experiments on both synthetic and real data.
arXiv Detail & Related papers (2021-08-06T12:59:12Z) - Effective and Efficient Computation with Multiple-timescale Spiking
Recurrent Neural Networks [0.9790524827475205]
We show how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance.
We calculate a $>$100x energy improvement for our SRNNs over classical RNNs on the harder tasks.
arXiv Detail & Related papers (2020-05-24T01:04:53Z) - A Hybrid Method for Training Convolutional Neural Networks [3.172761915061083]
We propose a hybrid method that uses both backpropagation and evolutionary strategies to train Convolutional Neural Networks.
We show that the proposed hybrid method is capable of improving upon regular training in the task of image classification.
arXiv Detail & Related papers (2020-04-15T17:52:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.