Stability of Internal States in Recurrent Neural Networks Trained on Regular Languages

• URL: http://arxiv.org/abs/2006.10828v1
• Date: Thu, 18 Jun 2020 19:50:15 GMT
• Title: Stability of Internal States in Recurrent Neural Networks Trained on Regular Languages
• Authors: Christian Oliva and Luis F. Lago-Fern\'andez
• Abstract summary: We study the stability of neural networks trained to recognize regular languages. In this saturated regime, analysis of the network activation shows a set of clusters that resemble discrete states in a finite state machine. We show that transitions between these states in response to input symbols are deterministic and stable.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We provide an empirical study of the stability of recurrent neural networks trained to recognize regular languages. When a small amount of noise is introduced into the activation function, the neurons in the recurrent layer tend to saturate in order to compensate the variability. In this saturated regime, analysis of the network activation shows a set of clusters that resemble discrete states in a finite state machine. We show that transitions between these states in response to input symbols are deterministic and stable. The networks display a stable behavior for arbitrarily long strings, and when random perturbations are applied to any of the states, they are able to recover and their evolution converges to the original clusters. This observation reinforces the interpretation of the networks as finite automata, with neurons or groups of neurons coding specific and meaningful input patterns.

Related papers

• Discovering Chunks in Neural Embeddings for Interpretability [53.80157905839065]
  We propose leveraging the principle of chunking to interpret artificial neural population activities. We first demonstrate this concept in recurrent neural networks (RNNs) trained on artificial sequences with imposed regularities. We identify similar recurring embedding states corresponding to concepts in the input, with perturbations to these states activating or inhibiting the associated concepts.
  arXiv  Detail & Related papers  (2025-02-03T20:30:46Z)
• Advancing Spatio-Temporal Processing in Spiking Neural Networks through Adaptation [6.233189707488025]
  In this article, we analyze the dynamical, computational, and learning properties of adaptive LIF neurons and networks thereof. We show that the superiority of networks of adaptive LIF neurons extends to the prediction and generation of complex time series.
  arXiv  Detail & Related papers  (2024-08-14T12:49:58Z)
• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Input correlations impede suppression of chaos and learning in balanced rate networks [58.720142291102135]
  Information encoding and learning in neural circuits depend on how well time-varying stimuli can control spontaneous network activity. We show that in firing-rate networks in the balanced state, external control of recurrent dynamics, strongly depends on correlations in the input.
  arXiv  Detail & Related papers  (2022-01-24T19:20:49Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• Lipschitz Recurrent Neural Networks [100.72827570987992]
  We show that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks.
  arXiv  Detail & Related papers  (2020-06-22T08:44:52Z)
• Separation of Memory and Processing in Dual Recurrent Neural Networks [0.0]
  We explore a neural network architecture that stacks a recurrent layer and a feedforward layer that is also connected to the input. When noise is introduced into the activation function of the recurrent units, these neurons are forced into a binary activation regime that makes the networks behave much as finite automata.
  arXiv  Detail & Related papers  (2020-05-17T11:38:42Z)
• Translating Diffusion, Wavelets, and Regularisation into Residual Networks [15.104201344012347]
  Convolutional neural networks (CNNs) often perform well, but their stability is poorly understood. We consider the simple problem of signal denoising, where classical approaches offer provable stability guarantees. We interpret numerical approximations of these classical methods as a specific residual network architecture. This leads to a dictionary which allows to translate diffusivities, shrinkage functions, and regularisers into activation functions.
  arXiv  Detail & Related papers  (2020-02-07T13:07:34Z)
• Mean-Field and Kinetic Descriptions of Neural Differential Equations [0.0]
  In this work we focus on a particular class of neural networks, i.e. the residual neural networks. We analyze steady states and sensitivity with respect to the parameters of the network, namely the weights and the bias. A modification of the microscopic dynamics, inspired by residual neural networks, leads to a Fokker-Planck formulation of the network.
  arXiv  Detail & Related papers  (2020-01-07T13:41:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.