Pruning Increases Orderedness in Recurrent Computation

• URL: http://arxiv.org/abs/2507.14747v1
• Date: Sat, 19 Jul 2025 20:44:17 GMT
• Title: Pruning Increases Orderedness in Recurrent Computation
• Authors: Yiding Song,
• Abstract summary: We investigate the degree to which directionality is a helpful inductive bias for artificial neural networks.<n>Taking directionality as topologically-ordered information flow between neurons, we formalise a perceptron layer with all-to-all connections.<n>We demonstrate that directionality can be induced rather than hard-wired by applying appropriate pruning techniques.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Inspired by the prevalence of recurrent circuits in biological brains, we investigate the degree to which directionality is a helpful inductive bias for artificial neural networks. Taking directionality as topologically-ordered information flow between neurons, we formalise a perceptron layer with all-to-all connections (mathematically equivalent to a weight-tied recurrent neural network) and demonstrate that directionality, a hallmark of modern feed-forward networks, can be induced rather than hard-wired by applying appropriate pruning techniques. Across different random seeds our pruning schemes successfully induce greater topological ordering in information flow between neurons without compromising performance, suggesting that directionality is not a prerequisite for learning, but may be an advantageous inductive bias discoverable by gradient descent and sparsification.

Related papers

• Evolution imposes an inductive bias that alters and accelerates learning dynamics [49.1574468325115]
  We investigate the effect of evolutionary optimization on the learning dynamics of neural networks.<n>We combined algorithms natural selection and online learning to produce a method for evolutionarily conditioning artificial neural networks.<n>Results suggest evolution constitutes an inductive bias that tunes neural systems to enable rapid learning.
  arXiv  Detail & Related papers  (2025-05-15T18:50:57Z)
• Correlations Are Ruining Your Gradient Descent [1.2432046687586285]
  Natural gradient descent illuminates how gradient vectors, pointing at directions of steepest descent, can be improved by considering the local curvature of loss landscapes. We show that correlations in the data at any linear transformation, including node responses at every layer of a neural network, cause a non-orthonormal relationship between the model's parameters. We describe a range of methods which have been proposed for decorrelation and whitening of node output, and expand on these to provide a novel method specifically useful for distributed computing and computational neuroscience.
  arXiv  Detail & Related papers  (2024-07-15T14:59:43Z)
• Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks [5.271584191900265]
  Training spiking neural networks to approximate universal functions is essential for studying information processing in the brain. The binary nature of spikes poses a challenge for direct gradient-based training. Here, we investigate the relation of surrogate gradients to two theoretically well-founded approaches.
  arXiv  Detail & Related papers  (2024-04-23T12:20:09Z)
• Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
  We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence. We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers. This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
  arXiv  Detail & Related papers  (2023-07-20T13:34:11Z)
• Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks [0.8932296777085644]
  Conventional artificial deep neural networks operating near the phase boundary of the signal propagation dynamics, also known as the edge of chaos, exhibit universal scaling laws of absorbing phase transitions.<n>We exploit the fully deterministic nature of the propagation dynamics to elucidate an analogy between a signal collapse in the neural networks and an absorbing state.
  arXiv  Detail & Related papers  (2023-07-05T13:39:02Z)
• Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry [43.584567991256925]
  We propose a new normative approach to describe the signal propagation in biological neural networks in both forward and backward directions. This framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm. Our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths.
  arXiv  Detail & Related papers  (2023-06-07T22:14:33Z)
• Contrastive-Signal-Dependent Plasticity: Self-Supervised Learning in Spiking Neural Circuits [61.94533459151743]
  This work addresses the challenge of designing neurobiologically-motivated schemes for adjusting the synapses of spiking networks. Our experimental simulations demonstrate a consistent advantage over other biologically-plausible approaches when training recurrent spiking networks.
  arXiv  Detail & Related papers  (2023-03-30T02:40:28Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.