Traveling Waves Integrate Spatial Information Through Time

• URL: http://arxiv.org/abs/2502.06034v3
• Date: Mon, 24 Feb 2025 16:47:01 GMT
• Title: Traveling Waves Integrate Spatial Information Through Time
• Authors: Mozes Jacobs, Roberto C. Budzinski, Lyle Muller, Demba Ba, T. Anderson Keller,
• Abstract summary: We introduce convolutional recurrent neural networks that learn to produce traveling waves in their hidden states in response to visual stimuli.<n>We observe that traveling waves effectively expand the receptive field of locally connected neurons, supporting long-range encoding and communication of information.<n>As a first step toward traveling-wave-based communication and visual representation in artificial networks, our findings suggest wave-dynamics may provide efficiency and training stability benefits.
• Score: 3.3496112914071166
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traveling waves of neural activity are widely observed in the brain, but their precise computational function remains unclear. One prominent hypothesis is that they enable the transfer and integration of spatial information across neural populations. However, few computational models have explored how traveling waves might be harnessed to perform such integrative processing. Drawing inspiration from the famous "Can one hear the shape of a drum?" problem -- which highlights how normal modes of wave dynamics encode geometric information -- we investigate whether similar principles can be leveraged in artificial neural networks. Specifically, we introduce convolutional recurrent neural networks that learn to produce traveling waves in their hidden states in response to visual stimuli, enabling spatial integration. By then treating these wave-like activation sequences as visual representations themselves, we obtain a powerful representational space that outperforms local feed-forward networks on tasks requiring global spatial context. In particular, we observe that traveling waves effectively expand the receptive field of locally connected neurons, supporting long-range encoding and communication of information. We demonstrate that models equipped with this mechanism solve visual semantic segmentation tasks demanding global integration, significantly outperforming local feed-forward models and rivaling non-local U-Net models with fewer parameters. As a first step toward traveling-wave-based communication and visual representation in artificial networks, our findings suggest wave-dynamics may provide efficiency and training stability benefits, while simultaneously offering a new framework for connecting models to biological recordings of neural activity.

Related papers

• Allostatic Control of Persistent States in Spiking Neural Networks for perception and computation [79.16635054977068]
  We introduce a novel model for updating perceptual beliefs about the environment by extending the concept of Allostasis to the control of internal representations. In this paper, we focus on an application in numerical cognition, where a bump of activity in an attractor network is used as a spatial numerical representation.
  arXiv  Detail & Related papers  (2025-03-20T12:28:08Z)
• Artificial Kuramoto Oscillatory Neurons [65.16453738828672]
  It has long been known in both neuroscience and AI that ''binding'' between neurons leads to a form of competitive learning. We introduce Artificial rethinking together with arbitrary connectivity designs such as fully connected convolutional, or attentive mechanisms. We show that this idea provides performance improvements across a wide spectrum of tasks such as unsupervised object discovery, adversarial robustness, uncertainty, and reasoning.
  arXiv  Detail & Related papers  (2024-10-17T17:47:54Z)
• Emergence of global receptive fields capturing multipartite quantum correlations [0.565473932498362]
  In quantum physics, even simple data with a well-defined structure at the wave function level can be characterized by extremely complex correlations. We show that monitoring the neural network weight space while learning quantum statistics allows to develop physical intuition about complex multipartite patterns. Our findings suggest a fresh look at constructing convolutional neural networks for processing data with non-local patterns.
  arXiv  Detail & Related papers  (2024-08-23T12:45:40Z)
• The Cooperative Network Architecture: Learning Structured Networks as Representation of Sensory Patterns [3.9848584845601014]
  We present the cooperative network architecture (CNA), a model that learns such net structure to represent input patterns and deals robustly with noise, deformation, and out-of-distribution data.
  arXiv  Detail & Related papers  (2024-07-08T06:22:10Z)
• Spiking representation learning for associative memories [0.0]
  We introduce a novel artificial spiking neural network (SNN) that performs unsupervised representation learning and associative memory operations. The architecture of our model derives from the neocortical columnar organization and combines feedforward projections for learning hidden representations and recurrent projections for forming associative memories.
  arXiv  Detail & Related papers  (2024-06-05T08:30:11Z)
• Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks [59.38765771221084]
  We present a physiologically inspired speech recognition architecture compatible and scalable with deep learning frameworks. We show end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronising neural activity to improve recognition performance.
  arXiv  Detail & Related papers  (2024-04-22T09:40:07Z)
• Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks [3.686808512438363]
  We leverage the concept of traveling wave dynamics within a neural lattice to formulate a theoretical model of neural working memory. We rigorously examine the model's capabilities in representing and learning state histories. Our findings suggest the broader relevance of traveling waves in AI and its potential in advancing neural network architectures.
  arXiv  Detail & Related papers  (2024-02-15T18:08:58Z)
• On the Trade-off Between Efficiency and Precision of Neural Abstraction [62.046646433536104]
  Neural abstractions have been recently introduced as formal approximations of complex, nonlinear dynamical models. We employ formal inductive synthesis procedures to generate neural abstractions that result in dynamical models with these semantics.
  arXiv  Detail & Related papers  (2023-07-28T13:22:32Z)
• Trainability, Expressivity and Interpretability in Gated Neural ODEs [0.0]
  We introduce a novel measure of expressivity which probes the capacity of a neural network to generate complex trajectories. We show how reduced-dimensional gnODEs retain their modeling power while greatly improving interpretability. We also demonstrate the benefit of gating in nODEs on several real-world tasks.
  arXiv  Detail & Related papers  (2023-07-12T18:29:01Z)
• Deep Representations for Time-varying Brain Datasets [4.129225533930966]
  This paper builds an efficient graph neural network model that incorporates both region-mapped fMRI sequences and structural connectivities as inputs. We find good representations of the latent brain dynamics through learning sample-level adaptive adjacency matrices. These modules can be easily adapted to and are potentially useful for other applications outside the neuroscience domain.
  arXiv  Detail & Related papers  (2022-05-23T21:57:31Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Can the brain use waves to solve planning problems? [62.997667081978825]
  We present a neural network model which can solve such tasks. The model is compatible with a broad range of empirical findings about the mammalian neocortex and hippocampus.
  arXiv  Detail & Related papers  (2021-10-11T11:07:05Z)
• Reservoir Memory Machines as Neural Computers [70.5993855765376]
  Differentiable neural computers extend artificial neural networks with an explicit memory without interference. We achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently.
  arXiv  Detail & Related papers  (2020-09-14T12:01:30Z)
• Wave Propagation of Visual Stimuli in Focus of Attention [77.4747032928547]
  Fast reactions to changes in the surrounding visual environment require efficient attention mechanisms to reallocate computational resources to most relevant locations in the visual field. We present a biologically-plausible model of focus of attention that exhibits effectiveness and efficiency exhibited by foveated animals.
  arXiv  Detail & Related papers  (2020-06-19T09:33:21Z)
• Implicit Neural Representations with Periodic Activation Functions [109.2353097792111]
  Implicitly defined, continuous, differentiable signal representations parameterized by neural networks have emerged as a powerful paradigm. We propose to leverage periodic activation functions for implicit neural representations and demonstrate that these networks, dubbed sinusoidal representation networks or Sirens, are ideally suited for representing complex natural signals and their derivatives. We show how Sirens can be leveraged to solve challenging boundary value problems, such as particular Eikonal equations, the Poisson equation, and the Helmholtz and wave equations.
  arXiv  Detail & Related papers  (2020-06-17T05:13:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.