Dynamics of specialization in neural modules under resource constraints

• URL: http://arxiv.org/abs/2106.02626v6
• Date: Sat, 12 Oct 2024 12:17:32 GMT
• Title: Dynamics of specialization in neural modules under resource constraints
• Authors: Gabriel Béna, Dan F. M. Goodman,
• Abstract summary: We use artificial neural networks to test the hypothesis that structural modularity is sufficient to guarantee functional specialization. We conclude that a static notion of specialization, based on structural modularity, is likely too simple a framework for understanding intelligence in situations of real-world complexity.
• Score: 2.9465623430708905
• License:
• Abstract: It has long been believed that the brain is highly modular both in terms of structure and function, although recent evidence has led some to question the extent of both types of modularity. We used artificial neural networks to test the hypothesis that structural modularity is sufficient to guarantee functional specialization, and find that in general, this doesn't necessarily hold. We then systematically tested which features of the environment and network do lead to the emergence of specialization. We used a simple toy environment, task and network, allowing us precise control, and show that in this setup, several distinct measures of specialization give qualitatively similar results. We further find that in this setup (1) specialization can only emerge in environments where features of that environment are meaningfully separable, (2) specialization preferentially emerges when the network is strongly resource-constrained, and (3) these findings are qualitatively similar across the different variations of network architectures that we tested, but that the quantitative relationships depend on the precise architecture. Finally, we show that functional specialization varies dynamically across time, and demonstrate that these dynamics depend on both the timing and bandwidth of information flow in the network. We conclude that a static notion of specialization, based on structural modularity, is likely too simple a framework for understanding intelligence in situations of real-world complexity, from biology to brain-inspired neuromorphic systems. We propose that thoroughly stress testing candidate definitions of functional modularity in simplified scenarios before extending to more complex data, network models and electrophysiological recordings is likely to be a fruitful approach.

Related papers

• Spatial embedding promotes a specific form of modularity with low entropy and heterogeneous spectral dynamics [0.0]
  Spatially embedded recurrent neural networks provide a promising avenue to study how modelled constraints shape the combined structural and functional organisation of networks over learning. We show that it is possible to study these restrictions through entropic measures of the neural weights and eigenspectrum, across both rate and spiking neural networks. This work deepens our understanding of constrained learning in neural networks, across coding schemes and tasks, where solutions to simultaneous structural and functional objectives must be accomplished in tandem.
  arXiv  Detail & Related papers  (2024-09-26T10:00:05Z)
• On The Specialization of Neural Modules [16.83151955540625]
  We study the ability of network modules to specialize to useful structures in a dataset and achieve systematic generalization. Our results shed light on the difficulty of module specialization, what is required for modules to successfully specialize, and the necessity of modular architectures to achieve systematicity.
  arXiv  Detail & Related papers  (2024-09-23T12:58:11Z)
• Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning [0.0]
  We show that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics. We demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales.
  arXiv  Detail & Related papers  (2024-06-10T13:44:07Z)
• Independent Modular Networks [3.10678167047537]
  Monolithic neural networks dismiss the compositional nature of data generation processes. We propose a modular network architecture that splits the modules into roles. We also provide regularizations that improve the resiliency of the modular network to the problem of module collapse.
  arXiv  Detail & Related papers  (2023-06-02T07:29:36Z)
• Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learning [2.038038953957366]
  We find that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment. We used a classic neuroevolutionary algorithm which enables rich, automatic optimisation and exploration of neural network architectures.
  arXiv  Detail & Related papers  (2022-05-13T05:18:18Z)
• Leveraging the structure of dynamical systems for data-driven modeling [111.45324708884813]
  We consider the impact of the training set and its structure on the quality of the long-term prediction. We show how an informed design of the training set, based on invariants of the system and the structure of the underlying attractor, significantly improves the resulting models.
  arXiv  Detail & Related papers  (2021-12-15T20:09:20Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Co-evolution of Functional Brain Network at Multiple Scales during Early Infancy [52.4179778122852]
  This paper leveraged a longitudinal infant resting-state functional magnetic resonance imaging dataset from birth to 2 years of age. By applying our proposed methodological framework on the collected longitudinal infant dataset, we provided the first evidence that, in the first 2 years of life, the brain functional network is co-evolved at different scales.
  arXiv  Detail & Related papers  (2020-09-15T07:21:04Z)
• Automated Search for Resource-Efficient Branched Multi-Task Networks [81.48051635183916]
  We propose a principled approach, rooted in differentiable neural architecture search, to automatically define branching structures in a multi-task neural network. We show that our approach consistently finds high-performing branching structures within limited resource budgets.
  arXiv  Detail & Related papers  (2020-08-24T09:49:19Z)
• S2RMs: Spatially Structured Recurrent Modules [105.0377129434636]
  We take a step towards exploiting dynamic structure that are capable of simultaneously exploiting both modular andtemporal structures. We find our models to be robust to the number of available views and better capable of generalization to novel tasks without additional training.
  arXiv  Detail & Related papers  (2020-07-13T17:44:30Z)
• Obtaining Faithful Interpretations from Compositional Neural Networks [72.41100663462191]
  We evaluate the intermediate outputs of NMNs on NLVR2 and DROP datasets. We find that the intermediate outputs differ from the expected output, illustrating that the network structure does not provide a faithful explanation of model behaviour.
  arXiv  Detail & Related papers  (2020-05-02T06:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.