Unsupervised Learning in Echo State Networks for Input Reconstruction

• URL: http://arxiv.org/abs/2501.11409v4
• Date: Wed, 30 Jul 2025 14:09:44 GMT
• Title: Unsupervised Learning in Echo State Networks for Input Reconstruction
• Authors: Taiki Yamada, Yuichi Katori, Kantaro Fujiwara,
• Abstract summary: Echo state networks (ESNs) are a class of recurrent neural networks in which only the readout layer is trainable.<n>In this study, we focus on input reconstruction (IR), where the readout layer is trained to reconstruct the input time series fed into the ESN.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Echo state networks (ESNs) are a class of recurrent neural networks in which only the readout layer is trainable, while the recurrent and input layers are fixed. This architectural constraint enables computationally efficient processing of time-series data. Traditionally, the readout layer in ESNs is trained using supervised learning with target outputs. In this study, we focus on input reconstruction (IR), where the readout layer is trained to reconstruct the input time series fed into the ESN. We show that IR can be achieved through unsupervised learning (UL), without access to supervised targets, provided that the ESN parameters are known a priori and satisfy invertibility conditions. This formulation allows applications relying on IR, such as dynamical system replication and noise filtering, to be reformulated within the UL framework via straightforward integration with existing algorithms. Our results suggest that prior knowledge of ESN parameters can reduce reliance on supervision, thereby establishing a new principle: not only by fixing part of the network parameters but also by exploiting their specific values. Furthermore, our UL-based algorithms for input reconstruction and related tasks are suitable for autonomous processing, offering insights into how analogous computational mechanisms might operate in the brain in principle. These findings contribute to a deeper understanding of the mathematical foundations of ESNs and their relevance to models in computational neuroscience.

Related papers

• Introducing Interval Neural Networks for Uncertainty-Aware System Identification [2.8391355909797644]
  This paper introduces a framework for constructing and learning Interval Neural Networks (INNs) to perform uncertainty quantification (UQ) in System Identification (SysID) tasks.<n>INNs are derived by transforming the learnable parameters (LPs) of pre-trained neural networks into interval-valued LPs without relying on probabilistic assumptions.<n>We extend Long Short-Term Memory (LSTM) and Neural Ordinary Differential Equations (Neural ODEs) into Interval LSTM (ILSTM) and Interval NODE (INODE) architectures, providing the mathematical foundations for their application in SysID.
  arXiv  Detail & Related papers  (2025-04-26T08:16:46Z)
• A Unified Framework for Neural Computation and Learning Over Time [56.44910327178975]
  Hamiltonian Learning is a novel unified framework for learning with neural networks "over time" It is based on differential equations that: (i) can be integrated without the need of external software solvers; (ii) generalize the well-established notion of gradient-based learning in feed-forward and recurrent networks; (iii) open to novel perspectives.
  arXiv  Detail & Related papers  (2024-09-18T14:57:13Z)
• Recurrent Stochastic Configuration Networks for Temporal Data Analytics [3.8719670789415925]
  This paper develops a recurrent version of configuration networks (RSCNs) for problem solving.<n>We build an initial RSCN model in the light of a supervisory mechanism, followed by an online update of the output weights.<n> Numerical results clearly indicate that the proposed RSCN performs favourably over all of the datasets.
  arXiv  Detail & Related papers  (2024-06-21T03:21:22Z)
• Improving the Performance of Echo State Networks Through State Feedback [0.0]
  Reservoir computing, using nonlinear dynamical systems, offers a cost-effective alternative to neural networks.<n>A potential drawback of ESNs is that the fixed reservoir may not offer the complexity needed for specific problems.<n>In this paper, we demonstrate that by feeding some component of the reservoir state back into the network through the input, we can drastically improve upon the performance of a given ESN.
  arXiv  Detail & Related papers  (2023-12-23T02:34:50Z)
• Deep Inductive Logic Programming meets Reinforcement Learning [0.0]
  Differentiable Neural Logic (dNL) networks are able to learn functions as their neural architecture includes symbolic reasoning. We propose an application of dNL in the field ofReinforcement Learning (RRL) to address dynamic continuous environments.
  arXiv  Detail & Related papers  (2023-08-30T09:08:46Z)
• Properties and Potential Applications of Random Functional-Linked Types of Neural Networks [81.56822938033119]
  Random functional-linked neural networks (RFLNNs) offer an alternative way of learning in deep structure. This paper gives some insights into the properties of RFLNNs from the viewpoints of frequency domain. We propose a method to generate a BLS network with better performance, and design an efficient algorithm for solving Poison's equation.
  arXiv  Detail & Related papers  (2023-04-03T13:25:22Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Fusing Event-based Camera and Radar for SLAM Using Spiking Neural Networks with Continual STDP Learning [7.667590910539249]
  This work proposes a first-of-its-kind SLAM architecture fusing an event-based camera and a Frequency Modulated Continuous Wave (FMCW) radar for drone navigation. Each sensor is processed by a bio-inspired Spiking Neural Network (SNN) with continual Spike-Timing-Dependent Plasticity (STDP) learning. We conduct numerous experiments to benchmark our system against state-of-the-art RGB methods and we demonstrate the robustness of our DVS-Radar SLAM approach.
  arXiv  Detail & Related papers  (2022-10-09T12:05:19Z)
• Less is More: Rethinking Few-Shot Learning and Recurrent Neural Nets [2.824895388993495]
  We provide theoretical guarantees for reliable learning under the information-theoretic AEP. We then focus on a highly efficient recurrent neural net (RNN) framework and propose a reduced-entropy algorithm for few-shot learning. Our experimental results demonstrate significant potential for improving learning models' sample efficiency, generalization, and time complexity.
  arXiv  Detail & Related papers  (2022-09-28T17:33:11Z)
• Composite FORCE learning of chaotic echo state networks for time-series prediction [7.650966670809372]
  This paper proposes a composite FORCE learning method to train ESNs whose initial activity is spontaneously chaotic. numerical results have shown that it significantly improves learning and prediction performances compared with existing methods.
  arXiv  Detail & Related papers  (2022-07-06T03:44:09Z)
• Pretraining Graph Neural Networks for few-shot Analog Circuit Modeling and Design [68.1682448368636]
  We present a supervised pretraining approach to learn circuit representations that can be adapted to new unseen topologies or unseen prediction tasks. To cope with the variable topological structure of different circuits we describe each circuit as a graph and use graph neural networks (GNNs) to learn node embeddings. We show that pretraining GNNs on prediction of output node voltages can encourage learning representations that can be adapted to new unseen topologies or prediction of new circuit level properties.
  arXiv  Detail & Related papers  (2022-03-29T21:18:47Z)
• Deep Q-network using reservoir computing with multi-layered readout [0.0]
  Recurrent neural network (RNN) based reinforcement learning (RL) is used for learning context-dependent tasks. An approach with replay memory introducing reservoir computing has been proposed, which trains an agent without BPTT. This paper shows that the performance of this method improves by using a multi-layered neural network for the readout layer.
  arXiv  Detail & Related papers  (2022-03-03T00:32:55Z)
• FF-NSL: Feed-Forward Neural-Symbolic Learner [70.978007919101]
  This paper introduces a neural-symbolic learning framework, called Feed-Forward Neural-Symbolic Learner (FF-NSL) FF-NSL integrates state-of-the-art ILP systems based on the Answer Set semantics, with neural networks, in order to learn interpretable hypotheses from labelled unstructured data.
  arXiv  Detail & Related papers  (2021-06-24T15:38:34Z)
• Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training [18.521882534906972]
  We propose to structure dropout patterns, by dropping out the same set of physical neurons within a batch, resulting in column (row) level hidden state sparsity. We conduct experiments for three representative NLP tasks: language modelling on the PTB dataset, OpenNMT based machine translation using the IWSLT De-En and En-Vi datasets, and named entity recognition sequence labelling.
  arXiv  Detail & Related papers  (2021-06-22T22:44:32Z)
• A SAR speckle filter based on Residual Convolutional Neural Networks [68.8204255655161]
  This work aims to present a novel method for filtering the speckle noise from Sentinel-1 data by applying Deep Learning (DL) algorithms, based on Convolutional Neural Networks (CNNs) The obtained results, if compared with the state of the art, show a clear improvement in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM)
  arXiv  Detail & Related papers  (2021-04-19T14:43:07Z)
• Reinforcement Learning with External Knowledge by using Logical Neural Networks [67.46162586940905]
  A recent neuro-symbolic framework called the Logical Neural Networks (LNNs) can simultaneously provide key-properties of both neural networks and symbolic logic. We propose an integrated method that enables model-free reinforcement learning from external knowledge sources.
  arXiv  Detail & Related papers  (2021-03-03T12:34:59Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)
• Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G Networks [84.2155885234293]
  We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC. To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
  arXiv  Detail & Related papers  (2020-02-22T14:38:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.