Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks

• URL: http://arxiv.org/abs/2010.02066v3
• Date: Sat, 6 Mar 2021 17:35:13 GMT
• Title: Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• Authors: R\'obert Csord\'as, Sjoerd van Steenkiste, J\"urgen Schmidhuber
• Abstract summary: Understanding if and how NNs are modular could provide insights into how to improve them. Current inspection methods, however, fail to link modules to their functionality.
• Score: 10.0444013205203
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks (NNs) whose subnetworks implement reusable functions are expected to offer numerous advantages, including compositionality through efficient recombination of functional building blocks, interpretability, preventing catastrophic interference, etc. Understanding if and how NNs are modular could provide insights into how to improve them. Current inspection methods, however, fail to link modules to their functionality. In this paper, we present a novel method based on learning binary weight masks to identify individual weights and subnets responsible for specific functions. Using this powerful tool, we contribute an extensive study of emerging modularity in NNs that covers several standard architectures and datasets. We demonstrate how common NNs fail to reuse submodules and offer new insights into the related issue of systematic generalization on language tasks.

Related papers

• Breaking Neural Network Scaling Laws with Modularity [8.482423139660153]
  We show how the amount of training data required to generalize varies with the intrinsic dimensionality of a task's input. We then develop a novel learning rule for modular networks to exploit this advantage.
  arXiv  Detail & Related papers  (2024-09-09T16:43:09Z)
• Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning [0.0]
  We show that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics. We demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales.
  arXiv  Detail & Related papers  (2024-06-10T13:44:07Z)
• OC-NMN: Object-centric Compositional Neural Module Network for Generative Visual Analogical Reasoning [49.12350554270196]
  We show how modularity can be leveraged to derive a compositional data augmentation framework inspired by imagination. Our method, denoted Object-centric Compositional Neural Module Network (OC-NMN), decomposes visual generative reasoning tasks into a series of primitives applied to objects without using a domain-specific language.
  arXiv  Detail & Related papers  (2023-10-28T20:12:58Z)
• Modular Deep Learning [120.36599591042908]
  Transfer learning has recently become the dominant paradigm of machine learning. It remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference. Modular deep learning has emerged as a promising solution to these challenges.
  arXiv  Detail & Related papers  (2023-02-22T18:11:25Z)
• Break It Down: Evidence for Structural Compositionality in Neural Networks [32.382094867951224]
  We show that neural networks can learn compositionality, obviating the need for specialized symbolic mechanisms. This suggests that neural networks may be able to learn compositionality, obviating the need for specialized symbolic mechanisms.
  arXiv  Detail & Related papers  (2023-01-26T00:53:11Z)
• Neural Attentive Circuits [93.95502541529115]
  We introduce a general purpose, yet modular neural architecture called Neural Attentive Circuits (NACs) NACs learn the parameterization and a sparse connectivity of neural modules without using domain knowledge. NACs achieve an 8x speedup at inference time while losing less than 3% performance.
  arXiv  Detail & Related papers  (2022-10-14T18:00:07Z)
• Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learning [2.038038953957366]
  We find that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment. We used a classic neuroevolutionary algorithm which enables rich, automatic optimisation and exploration of neural network architectures.
  arXiv  Detail & Related papers  (2022-05-13T05:18:18Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers [84.57980167400513]
  Neural Function Modules (NFM) aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm.
  arXiv  Detail & Related papers  (2020-10-15T20:43:17Z)
• Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models [61.480085460269514]
  We propose a framework for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models. We use this framework to build ModularQA, a system that can answer multi-hop reasoning questions by decomposing them into sub-questions answerable by a neural factoid single-span QA model and a symbolic calculator.
  arXiv  Detail & Related papers  (2020-09-01T23:45:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.