Hierarchical temporal receptive windows and zero-shot timescale generalization in biologically constrained scale-invariant deep networks

• URL: http://arxiv.org/abs/2601.02618v1
• Date: Tue, 06 Jan 2026 00:36:45 GMT
• Title: Hierarchical temporal receptive windows and zero-shot timescale generalization in biologically constrained scale-invariant deep networks
• Authors: Aakash Sarkar, Marc W. Howard,
• Abstract summary: We train biologically constrained deep networks based on scale-invariant hippocampal time cells.<n>We find that a hierarchy of TRWs emerged naturally across layers, despite the network having an identical spectrum of time constants within layers.<n>We then distilled these inductive priors into a biologically plausible recurrent architecture, SITH-RNN.
• Score: 1.2031796234206136
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human cognition integrates information across nested timescales. While the cortex exhibits hierarchical Temporal Receptive Windows (TRWs), local circuits often display heterogeneous time constants. To reconcile this, we trained biologically constrained deep networks, based on scale-invariant hippocampal time cells, on a language classification task mimicking the hierarchical structure of language (e.g., 'letters' forming 'words'). First, using a feedforward model (SITHCon), we found that a hierarchy of TRWs emerged naturally across layers, despite the network having an identical spectrum of time constants within layers. We then distilled these inductive priors into a biologically plausible recurrent architecture, SITH-RNN. Training a sequence of architectures ranging from generic RNNs to this restricted subset showed that the scale-invariant SITH-RNN learned faster with orders-of-magnitude fewer parameters, and generalized zero-shot to out-of-distribution timescales. These results suggest the brain employs scale-invariant, sequential priors - coding "what" happened "when" - making recurrent networks with such priors particularly well-suited to describe human cognition.

Related papers

• Bearing Syntactic Fruit with Stack-Augmented Neural Networks [59.49467149799849]
  We show that stack-augmented neural networks are more accurate models of human language acquisition than standard architectures.<n>We also propose a modification to the stack RNN architecture that improves hierarchical generalization.
  arXiv  Detail & Related papers  (2025-11-05T15:30:58Z)
• Temporal Chunking Enhances Recognition of Implicit Sequential Patterns [11.298233331771975]
  We propose a neuro-inspired approach that compresses temporal sequences into context-tagged chunks.<n>These tags are generated during an offline sleep phase and serve as compact references to past experience.<n>We evaluate this idea in a controlled synthetic environment designed to reveal the limitations of traditional neural network based sequence learners.
  arXiv  Detail & Related papers  (2025-05-31T14:51:08Z)
• The Role of Temporal Hierarchy in Spiking Neural Networks [2.0881857682885836]
  Spiking Neural Networks (SNNs) have the potential for rich-temporal signal processing thanks to exploiting both spatial and temporal parameters. Time constants have been recently shown to have computational benefits that help reduce the overall number of parameters required in the network. To reduce the cost of optimization, architectural biases can be applied, in this case in the temporal domain. We propose to impose a hierarchy of temporal representation in the hidden layers of SNNs, highlighting that such an inductive bias improves their performance.
  arXiv  Detail & Related papers  (2024-07-26T16:00:20Z)
• On The Expressivity of Recurrent Neural Cascades [48.87943990557107]
  Recurrent Neural Cascades (RNCs) are the recurrent neural networks with no cyclic dependencies among recurrent neurons. We show that RNCs can achieve the expressivity of all regular languages by introducing neurons that can implement groups.
  arXiv  Detail & Related papers  (2023-12-14T15:47:26Z)
• Time Regularization in Optimal Time Variable Learning [0.4490343701046724]
  Recently, optimal time variable learning in deep neural networks (DNNs) was introduced in arXiv:2204.08528. We extend the concept by introducing a regularization term that directly relates to the time horizon in discrete dynamical systems. We propose an adaptive pruning approach for Residual Neural Networks (ResNets) Results are illustrated by applying the proposed concepts to classification tasks on the well known MNIST and Fashion MNIST data sets.
  arXiv  Detail & Related papers  (2023-06-28T11:27:48Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification [53.55504611255664]
  FormerTime is a hierarchical representation model for improving the classification capacity for the multivariate time series classification task. It exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism.
  arXiv  Detail & Related papers  (2023-02-20T07:46:14Z)
• Generating fine-grained surrogate temporal networks [12.7211231166069]
  We propose a novel and simple method for generating surrogate temporal networks. Our method decomposes the input network into star-like structures evolving in time. Then those structures are used as building blocks to generate a surrogate temporal network.
  arXiv  Detail & Related papers  (2022-05-18T09:38:22Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• Temporal and Object Quantification Networks [95.64650820186706]
  We present a new class of neuro-symbolic networks with a structural bias that enables them to learn to recognize complex relational-temporal events. We demonstrate that TOQ-Nets can generalize from small amounts of data to scenarios containing more objects than were present during training and to temporal warpings of input sequences.
  arXiv  Detail & Related papers  (2021-06-10T16:18:21Z)
• DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales [0.0]
  Neural networks are either plagued by the exploding/vanishing gradient problem in recurrent neural networks (RNNs) or must adjust their parameters to learn the relevant time scales (e.g., in LSTMs) This paper introduces DeepSITH, a network comprising biologically-inspired Scale-Invariant Temporal History (SITH) modules in series with dense connections between layers. SITH modules respond to their inputs with a geometrically-spaced set of time constants, enabling the DeepSITH network to learn problems along a continuum of time-scales.
  arXiv  Detail & Related papers  (2021-04-09T23:58:14Z)
• PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning [109.84770951839289]
  We present PredRNN, a new recurrent network for learning visual dynamics from historical context. We show that our approach obtains highly competitive results on three standard datasets.
  arXiv  Detail & Related papers  (2021-03-17T08:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.