Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning

• URL: http://arxiv.org/abs/2406.06262v1
• Date: Mon, 10 Jun 2024 13:44:07 GMT
• Title: Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning
• Authors: Mani Hamidi, Sina Khajehabdollahi, Emmanouil Giannakakis, Tim Schäfer, Anna Levina, Charley M. Wu,
• Abstract summary: We show that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics. We demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Structural modularity is a pervasive feature of biological neural networks, which have been linked to several functional and computational advantages. Yet, the use of modular architectures in artificial neural networks has been relatively limited despite early successes. Here, we explore the performance and functional dynamics of a modular network trained on a memory task via an iterative growth curriculum. We find that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics, including training time, generalizability, and robustness to some perturbations. We further examine how different aspects of a modular network's connectivity contribute to its computational capability. We then demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed and only the connections between modules are trained. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales, and help build more scalable and compressible artificial networks.

Related papers

• Breaking Neural Network Scaling Laws with Modularity [8.482423139660153]
  We show how the amount of training data required to generalize varies with the intrinsic dimensionality of a task's input. We then develop a novel learning rule for modular networks to exploit this advantage.
  arXiv  Detail & Related papers  (2024-09-09T16:43:09Z)
• Growing Brains: Co-emergence of Anatomical and Functional Modularity in Recurrent Neural Networks [18.375521792153112]
  Recurrent neural networks (RNNs) trained on compositional tasks can exhibit functional modularity. We apply a recent machine learning method, brain-inspired modular training, to a network being trained to solve a set of compositional cognitive tasks. We find that functional and anatomical clustering emerge together, such that functionally similar neurons also become spatially localized and interconnected.
  arXiv  Detail & Related papers  (2023-10-11T17:58:25Z)
• Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One [60.5818387068983]
  Graph neural networks (GNN) suffer from severe inefficiency. We propose to decouple a multi-layer GNN as multiple simple modules for more efficient training. We show that the proposed framework is highly efficient with reasonable performance.
  arXiv  Detail & Related papers  (2023-04-20T07:21:32Z)
• Modular Deep Learning [120.36599591042908]
  Transfer learning has recently become the dominant paradigm of machine learning. It remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference. Modular deep learning has emerged as a promising solution to these challenges.
  arXiv  Detail & Related papers  (2023-02-22T18:11:25Z)
• Neural Attentive Circuits [93.95502541529115]
  We introduce a general purpose, yet modular neural architecture called Neural Attentive Circuits (NACs) NACs learn the parameterization and a sparse connectivity of neural modules without using domain knowledge. NACs achieve an 8x speedup at inference time while losing less than 3% performance.
  arXiv  Detail & Related papers  (2022-10-14T18:00:07Z)
• Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learning [2.038038953957366]
  We find that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment. We used a classic neuroevolutionary algorithm which enables rich, automatic optimisation and exploration of neural network architectures.
  arXiv  Detail & Related papers  (2022-05-13T05:18:18Z)
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers [84.57980167400513]
  Neural Function Modules (NFM) aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm.
  arXiv  Detail & Related papers  (2020-10-15T20:43:17Z)
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks [10.0444013205203]
  Understanding if and how NNs are modular could provide insights into how to improve them. Current inspection methods, however, fail to link modules to their functionality.
  arXiv  Detail & Related papers  (2020-10-05T15:04:11Z)
• Reservoir Memory Machines as Neural Computers [70.5993855765376]
  Differentiable neural computers extend artificial neural networks with an explicit memory without interference. We achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently.
  arXiv  Detail & Related papers  (2020-09-14T12:01:30Z)
• Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory [79.42778415729475]
  We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning. We show how to extend the architecture of a simple RNN by separating its hidden state into different modules. We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
  arXiv  Detail & Related papers  (2020-06-29T08:35:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.