On the Dynamics of Learning Time-Aware Behavior with Recurrent Neural Networks

• URL: http://arxiv.org/abs/2306.07125v1
• Date: Mon, 12 Jun 2023 14:01:30 GMT
• Title: On the Dynamics of Learning Time-Aware Behavior with Recurrent Neural Networks
• Authors: Peter DelMastro, Rushiv Arora, Edward Rietman, Hava T. Siegelmann
• Abstract summary: We introduce a family of supervised learning tasks dependent on hidden temporal variables. We train RNNs to emulate temporal flipflops that emphasize the need for time-awareness over long-term memory. We show that these RNNs learn to switch between periodic orbits that encode time modulo the period of the transition rules.
• Score: 2.294014185517203
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recurrent Neural Networks (RNNs) have shown great success in modeling time-dependent patterns, but there is limited research on their learned representations of latent temporal features and the emergence of these representations during training. To address this gap, we use timed automata (TA) to introduce a family of supervised learning tasks modeling behavior dependent on hidden temporal variables whose complexity is directly controllable. Building upon past studies from the perspective of dynamical systems, we train RNNs to emulate temporal flipflops, a new collection of TA that emphasizes the need for time-awareness over long-term memory. We find that these RNNs learn in phases: they quickly perfect any time-independent behavior, but they initially struggle to discover the hidden time-dependent features. In the case of periodic "time-of-day" aware automata, we show that the RNNs learn to switch between periodic orbits that encode time modulo the period of the transition rules. We subsequently apply fixed point stability analysis to monitor changes in the RNN dynamics during training, and we observe that the learning phases are separated by a bifurcation from which the periodic behavior emerges. In this way, we demonstrate how dynamical systems theory can provide insights into not only the learned representations of these models, but also the dynamics of the learning process itself. We argue that this style of analysis may provide insights into the training pathologies of recurrent architectures in contexts outside of time-awareness.

Related papers

• LINOCS: Lookahead Inference of Networked Operators for Continuous Stability [4.508868068781057]
  We introduce Lookahead-driven Inference of Networked Operators for Continuous Stability (LINOCS) LINOCS is a robust learning procedure for identifying hidden dynamical interactions in noisy time-series data. We demonstrate LINOCS' ability to recover the ground truth dynamical operators underlying synthetic time-series data.
  arXiv  Detail & Related papers  (2024-04-28T18:16:58Z)
• ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks [1.5931140598271163]
  We build a model for efficient learning sequences using only local always-on and phase-free plasticity. We showcase the capabilities of ELiSe in a mock-up of birdsong learning, and demonstrate its flexibility with respect to parametrization.
  arXiv  Detail & Related papers  (2024-02-26T17:30:34Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Brain-Inspired Spiking Neural Network for Online Unsupervised Time Series Prediction [13.521272923545409]
  We present a novel Continuous Learning-based Unsupervised Recurrent Spiking Neural Network Model (CLURSNN) CLURSNN makes online predictions by reconstructing the underlying dynamical system using Random Delay Embedding. We show that the proposed online time series prediction methodology outperforms state-of-the-art DNN models when predicting an evolving Lorenz63 dynamical system.
  arXiv  Detail & Related papers  (2023-04-10T16:18:37Z)
• Critical Learning Periods for Multisensory Integration in Deep Networks [112.40005682521638]
  We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training. We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations.
  arXiv  Detail & Related papers  (2022-10-06T23:50:38Z)
• Variational Predictive Routing with Nested Subjective Timescales [1.6114012813668934]
  We present Variational Predictive Routing (PRV) - a neural inference system that organizes latent video features in a temporal hierarchy. We show that VPR is able to detect event boundaries, disentangletemporal features, adapt to the dynamics hierarchy of the data, and produce accurate time-agnostic rollouts of the future.
  arXiv  Detail & Related papers  (2021-10-21T16:12:59Z)
• Causal Navigation by Continuous-time Neural Networks [108.84958284162857]
  We propose a theoretical and experimental framework for learning causal representations using continuous-time neural networks. We evaluate our method in the context of visual-control learning of drones over a series of complex tasks.
  arXiv  Detail & Related papers  (2021-06-15T17:45:32Z)
• Learning Temporal Dynamics from Cycles in Narrated Video [85.89096034281694]
  We propose a self-supervised solution to the problem of learning to model how the world changes as time elapses. Our model learns modality-agnostic functions to predict forward and backward in time, which must undo each other when composed. We apply the learned dynamics model without further training to various tasks, such as predicting future action and temporally ordering sets of images.
  arXiv  Detail & Related papers  (2021-01-07T02:41:32Z)
• Time-Reversal Symmetric ODE Network [138.02741983098454]
  Time-reversal symmetry is a fundamental property that frequently holds in classical and quantum mechanics. We propose a novel loss function that measures how well our ordinary differential equation (ODE) networks comply with this time-reversal symmetry. We show that, even for systems that do not possess the full time-reversal symmetry, TRS-ODENs can achieve better predictive performances over baselines.
  arXiv  Detail & Related papers  (2020-07-22T12:19:40Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.