Techniques for Enhancing Memory Capacity of Reservoir Computing

• URL: http://arxiv.org/abs/2502.17923v1
• Date: Tue, 25 Feb 2025 07:38:58 GMT
• Title: Techniques for Enhancing Memory Capacity of Reservoir Computing
• Authors: Atsuki Yokota, Ichiro Kawashima, Yohei Saito, Hakaru Tamukoh, Osamu Nomura, Takashi Morie,
• Abstract summary: Reservoir Computing (RC) is a bio-inspired machine learning framework.<n>There is a trade-off between memory capacity and nonlinearity in RC.<n>We propose methods to improve the memory capacity of reservoir models.
• Score: 0.7421845364041001
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reservoir Computing (RC) is a bio-inspired machine learning framework, and various models have been proposed. RC is a well-suited model for time series data processing, but there is a trade-off between memory capacity and nonlinearity. In this study, we propose methods to improve the memory capacity of reservoir models by modifying their network configuration except for the inside of reservoirs. The Delay method retains past inputs by adding delay node chains to the input layer with the specified number of delay steps. To suppress the effect of input value increase due to the Delay method, we divide the input weights by the number of added delay steps. The Pass through method feeds input values directly to the output layer. The Clustering method divides the input and reservoir nodes into multiple parts and integrates them at the output layer. We applied these methods to an echo state network (ESN), a typical RC model, and the chaotic Boltzmann machine (CBM)-RC, which can be efficiently implemented in integrated circuits. We evaluated their performance on the NARMA task, and measured information processing capacity (IPC) to evaluate the trade-off between memory capacity and nonlinearity.

Related papers

• Hardware-Friendly Delayed-Feedback Reservoir for Multivariate Time-Series Classification [3.8861692690368823]
  Reservoir computing is attracting attention as a machine-learning technique for edge computing. In time-series classification tasks, the number of features obtained using a reservoir depends on the length of the input series. We propose a dot-product-based reservoir representation (DPR) for RC based on the dot product of data features.
  arXiv  Detail & Related papers  (2025-04-16T11:22:38Z)
• SDP: Spiking Diffusion Policy for Robotic Manipulation with Learnable Channel-Wise Membrane Thresholds [7.4357764462464635]
  This paper introduces a Spiking Diffusion Policy (SDP) learning method for robotic manipulation. SDP integrates Spiking Neurons and Learnable Channel-wise Membrane Thresholds (LCMT) into the diffusion policy model. We achieve results comparable to those of the ANN counterparts, along with faster convergence speeds than the baseline SNN method.
  arXiv  Detail & Related papers  (2024-09-17T13:53:36Z)
• Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching [56.286064975443026]
  We make an interesting and somehow surprising observation: the computation of a large proportion of layers in the diffusion transformer, through a caching mechanism, can be readily removed even without updating the model parameters. We introduce a novel scheme, named Learningto-Cache (L2C), that learns to conduct caching in a dynamic manner for diffusion transformers. Experimental results show that L2C largely outperforms samplers such as DDIM and DPM-r, alongside prior cache-based methods at the same inference speed.
  arXiv  Detail & Related papers  (2024-06-03T18:49:57Z)
• Resistive Memory-based Neural Differential Equation Solver for Score-based Diffusion Model [55.116403765330084]
  Current AIGC methods, such as score-based diffusion, are still deficient in terms of rapidity and efficiency. We propose a time-continuous and analog in-memory neural differential equation solver for score-based diffusion. We experimentally validate our solution with 180 nm resistive memory in-memory computing macros.
  arXiv  Detail & Related papers  (2024-04-08T16:34:35Z)
• ADC/DAC-Free Analog Acceleration of Deep Neural Networks with Frequency Transformation [2.7488316163114823]
  This paper proposes a novel approach to an energy-efficient acceleration of frequency-domain neural networks by utilizing analog-domain frequency-based tensor transformations. Our approach achieves more compact cells by eliminating the need for trainable parameters in the transformation matrix. On a 16$times$16 crossbars, for 8-bit input processing, the proposed approach achieves the energy efficiency of 1602 tera operations per second per Watt.
  arXiv  Detail & Related papers  (2023-09-04T19:19:39Z)
• Memory Efficient Neural Processes via Constant Memory Attention Block [55.82269384896986]
  Constant Memory Attentive Neural Processes (CMANPs) are an NP variant that only requires constant memory. We show CMANPs achieve state-of-the-art results on popular NP benchmarks while being significantly more memory efficient than prior methods.
  arXiv  Detail & Related papers  (2023-05-23T23:10:19Z)
• Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance [68.8204255655161]
  We introduce a quantization-aware training algorithm that guarantees avoiding numerical overflow when reducing the precision of accumulators during inference. We evaluate our algorithm across multiple quantized models that we train for different tasks, showing that our approach can reduce the precision of accumulators while maintaining model accuracy with respect to a floating-point baseline.
  arXiv  Detail & Related papers  (2023-01-31T02:46:57Z)
• Latent Iterative Refinement for Modular Source Separation [44.78689915209527]
  Traditional source separation approaches train deep neural network models end-to-end with all the data available at once. We argue that we can significantly increase resource efficiency during both training and inference stages.
  arXiv  Detail & Related papers  (2022-11-22T00:02:57Z)
• Self-Gated Memory Recurrent Network for Efficient Scalable HDR Deghosting [59.04604001936661]
  We propose a novel recurrent network-based HDR deghosting method for fusing arbitrary length dynamic sequences. We introduce a new recurrent cell architecture, namely Self-Gated Memory (SGM) cell, that outperforms the standard LSTM cell. The proposed approach achieves state-of-the-art performance compared to existing HDR deghosting methods quantitatively across three publicly available datasets.
  arXiv  Detail & Related papers  (2021-12-24T12:36:33Z)
• Learning to Perform Downlink Channel Estimation in Massive MIMO Systems [72.76968022465469]
  We study downlink (DL) channel estimation in a Massive multiple-input multiple-output (MIMO) system. A common approach is to use the mean value as the estimate, motivated by channel hardening. We propose two novel estimation methods.
  arXiv  Detail & Related papers  (2021-09-06T13:42:32Z)
• Model-Size Reduction for Reservoir Computing by Concatenating Internal States Through Time [2.6872737601772956]
  Reservoir computing (RC) is a machine learning algorithm that can learn complex time series from data very rapidly. To implement RC in edge computing, it is highly important to reduce the amount of computational resources that RC requires. We propose methods that reduce the size of the reservoir by inputting the past or drifting states of the reservoir to the output layer at the current time step.
  arXiv  Detail & Related papers  (2020-06-11T06:11:03Z)
• Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations [143.3053365553897]
  We describe a procedure for removing dependency on a cohort of training data from a trained deep network. We introduce a new bound on how much information can be extracted per query about the forgotten cohort. We exploit the connections between the activation and weight dynamics of a DNN inspired by Neural Tangent Kernels to compute the information in the activations.
  arXiv  Detail & Related papers  (2020-03-05T23:17:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.