Exploiting Multiple Timescales in Hierarchical Echo State Networks
- URL: http://arxiv.org/abs/2101.04223v2
- Date: Thu, 14 Jan 2021 17:39:59 GMT
- Title: Exploiting Multiple Timescales in Hierarchical Echo State Networks
- Authors: Luca Manneschi, Matthew O. A. Ellis, Guido Gigante, Andrew C. Lin,
Paolo Del Giudice, Eleni Vasilaki
- Abstract summary: Echo state networks (ESNs) are a powerful form of reservoir computing that only require training of linear output weights.
Here we explore the timescales in hierarchical ESNs, where the reservoir is partitioned into two smaller reservoirs linked with distinct properties.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Echo state networks (ESNs) are a powerful form of reservoir computing that
only require training of linear output weights whilst the internal reservoir is
formed of fixed randomly connected neurons. With a correctly scaled
connectivity matrix, the neurons' activity exhibits the echo-state property and
responds to the input dynamics with certain timescales. Tuning the timescales
of the network can be necessary for treating certain tasks, and some
environments require multiple timescales for an efficient representation. Here
we explore the timescales in hierarchical ESNs, where the reservoir is
partitioned into two smaller linked reservoirs with distinct properties. Over
three different tasks (NARMA10, a reconstruction task in a volatile
environment, and psMNIST), we show that by selecting the hyper-parameters of
each partition such that they focus on different timescales, we achieve a
significant performance improvement over a single ESN. Through a linear
analysis, and under the assumption that the timescales of the first partition
are much shorter than the second's (typically corresponding to optimal
operating conditions), we interpret the feedforward coupling of the partitions
in terms of an effective representation of the input signal, provided by the
first partition to the second, whereby the instantaneous input signal is
expanded into a weighted combination of its time derivatives. Furthermore, we
propose a data-driven approach to optimise the hyper-parameters through a
gradient descent optimisation method that is an online approximation of
backpropagation through time. We demonstrate the application of the online
learning rule across all the tasks considered.
Related papers
- SineNet: Learning Temporal Dynamics in Time-Dependent Partial Differential Equations [46.244771659503314]
We propose SineNet, consisting of multiple sequentially connected U-shaped network blocks, referred to as waves.
In SineNet, high-resolution features are evolved progressively through multiple stages, thereby reducing the amount of misalignment within each stage.
Our method is rigorously tested on multiple PDE datasets, including the Navier-Stokes equations and shallow water equations.
arXiv Detail & Related papers (2024-03-28T15:41:41Z) - Temporal Aggregation and Propagation Graph Neural Networks for Dynamic
Representation [67.26422477327179]
Temporal graphs exhibit dynamic interactions between nodes over continuous time.
We propose a novel method of temporal graph convolution with the whole neighborhood.
Our proposed TAP-GNN outperforms existing temporal graph methods by a large margin in terms of both predictive performance and online inference latency.
arXiv Detail & Related papers (2023-04-15T08:17:18Z) - Efficient Parametric Approximations of Neural Network Function Space
Distance [6.117371161379209]
It is often useful to compactly summarize important properties of model parameters and training data so that they can be used later without storing and/or iterating over the entire dataset.
We consider estimating the Function Space Distance (FSD) over a training set, i.e. the average discrepancy between the outputs of two neural networks.
We propose a Linearized Activation TRick (LAFTR) and derive an efficient approximation to FSD for ReLU neural networks.
arXiv Detail & Related papers (2023-02-07T15:09:23Z) - UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation [93.88170217725805]
We propose a 3D medical image segmentation approach, named UNETR++, that offers both high-quality segmentation masks as well as efficiency in terms of parameters, compute cost, and inference speed.
The core of our design is the introduction of a novel efficient paired attention (EPA) block that efficiently learns spatial and channel-wise discriminative features.
Our evaluations on five benchmarks, Synapse, BTCV, ACDC, BRaTs, and Decathlon-Lung, reveal the effectiveness of our contributions in terms of both efficiency and accuracy.
arXiv Detail & Related papers (2022-12-08T18:59:57Z) - ZoPE: A Fast Optimizer for ReLU Networks with Low-Dimensional Inputs [30.34898838361206]
We present an algorithm called ZoPE that solves optimization problems over the output of feedforward ReLU networks with low-dimensional inputs.
Using ZoPE, we observe a $25times speedup on property 1 of the ACAS Xu neural network verification benchmark and an $85times speedup on a set of linear optimization problems.
arXiv Detail & Related papers (2021-06-09T18:36:41Z) - Multi-Time-Scale Input Approaches for Hourly-Scale Rainfall-Runoff
Modeling based on Recurrent Neural Networks [0.0]
Two approaches are proposed to reduce the required computational time for time-series modeling through a recurrent neural network (RNN)
One approach provides coarse fine temporal resolutions of the input time-series to RNN in parallel.
The results confirm that both of the proposed approaches can reduce the computational time for the training of RNN significantly.
arXiv Detail & Related papers (2021-01-30T07:51:55Z) - Deep Cellular Recurrent Network for Efficient Analysis of Time-Series
Data with Spatial Information [52.635997570873194]
This work proposes a novel deep cellular recurrent neural network (DCRNN) architecture to process complex multi-dimensional time series data with spatial information.
The proposed architecture achieves state-of-the-art performance while utilizing substantially less trainable parameters when compared to comparable methods in the literature.
arXiv Detail & Related papers (2021-01-12T20:08:18Z) - DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic
Convolution [136.7261709896713]
We propose a data-driven approach that generates the appropriate convolution kernels to apply in response to the nature of the instances.
The proposed method achieves promising results on both ScanetNetV2 and S3DIS.
It also improves inference speed by more than 25% over the current state-of-the-art.
arXiv Detail & Related papers (2020-11-26T14:56:57Z) - Communication-Efficient Distributed Stochastic AUC Maximization with
Deep Neural Networks [50.42141893913188]
We study a distributed variable for large-scale AUC for a neural network as with a deep neural network.
Our model requires a much less number of communication rounds and still a number of communication rounds in theory.
Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
arXiv Detail & Related papers (2020-05-05T18:08:23Z) - PairNets: Novel Fast Shallow Artificial Neural Networks on Partitioned
Subspaces [0.0]
We create a novel shallow 4-layer ANN called "Pairwise Neural Network" ("PairNet")
A value of each input is partitioned into multiple intervals, and then an n-dimensional space is partitioned into M n-dimensional subspaces.
M local PairNets are built in M partitioned local n-dimensional subspaces.
arXiv Detail & Related papers (2020-01-24T05:23:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.