Neural Sculpting: Uncovering hierarchically modular task structure in neural networks through pruning and network analysis

• URL: http://arxiv.org/abs/2305.18402v3
• Date: Fri, 27 Oct 2023 19:44:41 GMT
• Title: Neural Sculpting: Uncovering hierarchically modular task structure in neural networks through pruning and network analysis
• Authors: Shreyas Malakarjun Patil, Loizos Michael, Constantine Dovrolis
• Abstract summary: We show that hierarchically modular neural networks offer benefits such as learning efficiency, generalization, multi-task learning, and transfer. We propose an approach based on iterative unit and edge pruning (during training), combined with network analysis for module detection and hierarchy inference.
• Score: 8.080026425139708
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural target functions and tasks typically exhibit hierarchical modularity -- they can be broken down into simpler sub-functions that are organized in a hierarchy. Such sub-functions have two important features: they have a distinct set of inputs (input-separability) and they are reused as inputs higher in the hierarchy (reusability). Previous studies have established that hierarchically modular neural networks, which are inherently sparse, offer benefits such as learning efficiency, generalization, multi-task learning, and transfer. However, identifying the underlying sub-functions and their hierarchical structure for a given task can be challenging. The high-level question in this work is: if we learn a task using a sufficiently deep neural network, how can we uncover the underlying hierarchy of sub-functions in that task? As a starting point, we examine the domain of Boolean functions, where it is easier to determine whether a task is hierarchically modular. We propose an approach based on iterative unit and edge pruning (during training), combined with network analysis for module detection and hierarchy inference. Finally, we demonstrate that this method can uncover the hierarchical modularity of a wide range of Boolean functions and two vision tasks based on the MNIST digits dataset.

Related papers

• Feature Network Methods in Machine Learning and Applications [0.0]
  A machine learning (ML) feature network is a graph that connects ML features in learning tasks based on their similarity. We provide an example of a deep tree-structured feature network, where hierarchical connections are formed through feature clustering and feed-forward learning.
  arXiv  Detail & Related papers  (2024-01-10T01:57:12Z)
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model [47.617093812158366]
  We introduce the Random Hierarchy Model: a family of synthetic tasks inspired by the hierarchical structure of language and images. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Our results indicate how deep networks overcome the curse of dimensionality by building invariant representations.
  arXiv  Detail & Related papers  (2023-07-05T09:11:09Z)
• Dynamic Perceiver for Efficient Visual Recognition [87.08210214417309]
  We propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task. A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks. Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features.
  arXiv  Detail & Related papers  (2023-06-20T03:00:22Z)
• Break It Down: Evidence for Structural Compositionality in Neural Networks [32.382094867951224]
  We show that neural networks can learn compositionality, obviating the need for specialized symbolic mechanisms. This suggests that neural networks may be able to learn compositionality, obviating the need for specialized symbolic mechanisms.
  arXiv  Detail & Related papers  (2023-01-26T00:53:11Z)
• Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization [101.72755769194677]
  We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph. Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks. Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
  arXiv  Detail & Related papers  (2022-05-25T10:44:25Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Learning Task Decomposition with Ordered Memory Policy Network [73.3813423684999]
  We propose Ordered Memory Policy Network (OMPN) to discover subtask hierarchy by learning from demonstration. OMPN can be applied to partially observable environments and still achieve higher task decomposition performance. Our visualization confirms that the subtask hierarchy can emerge in our model.
  arXiv  Detail & Related papers  (2021-03-19T18:13:35Z)
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers [84.57980167400513]
  Neural Function Modules (NFM) aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm.
  arXiv  Detail & Related papers  (2020-10-15T20:43:17Z)
• Hierarchical community structure in networks [1.9981375888949475]
  We present a theoretical study on hierarchical community structure in networks. We address the following questions: 1) How should we define a hierarchy of communities? 2) How do we determine if there is sufficient evidence of a hierarchical structure in a network?
  arXiv  Detail & Related papers  (2020-09-15T16:18:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.