Modularity in Deep Learning: A Survey

• URL: http://arxiv.org/abs/2310.01154v1
• Date: Mon, 2 Oct 2023 12:41:34 GMT
• Title: Modularity in Deep Learning: A Survey
• Authors: Haozhe Sun (LISN, TAU, Inria), Isabelle Guyon (TAU, LISN, Inria)
• Abstract summary: We review the notion of modularity in deep learning around three axes: data, task, and model. Data modularity refers to the observation or creation of data groups for various purposes. Task modularity refers to the decomposition of tasks into sub-tasks. Model modularity means that the architecture of a neural network system can be decomposed into identifiable modules.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modularity is a general principle present in many fields. It offers attractive advantages, including, among others, ease of conceptualization, interpretability, scalability, module combinability, and module reusability. The deep learning community has long sought to take inspiration from the modularity principle, either implicitly or explicitly. This interest has been increasing over recent years. We review the notion of modularity in deep learning around three axes: data, task, and model, which characterize the life cycle of deep learning. Data modularity refers to the observation or creation of data groups for various purposes. Task modularity refers to the decomposition of tasks into sub-tasks. Model modularity means that the architecture of a neural network system can be decomposed into identifiable modules. We describe different instantiations of the modularity principle, and we contextualize their advantages in different deep learning sub-fields. Finally, we conclude the paper with a discussion of the definition of modularity and directions for future research.

Related papers

• Configurable Foundation Models: Building LLMs from a Modular Perspective [115.63847606634268]
  A growing tendency to decompose LLMs into numerous functional modules allows for inference with part of modules and dynamic assembly of modules to tackle complex tasks. We coin the term brick to represent each functional module, designating the modularized structure as customizable foundation models. We present four brick-oriented operations: retrieval and routing, merging, updating, and growing. We find that the FFN layers follow modular patterns with functional specialization of neurons and functional neuron partitions.
  arXiv  Detail & Related papers  (2024-09-04T17:01:02Z)
• Is Modularity Transferable? A Case Study through the Lens of Knowledge Distillation [59.37775534633868]
  We present an extremely straightforward approach to transferring pre-trained, task-specific PEFT modules between same-family PLMs. We also propose a method that allows the transfer of modules between incompatible PLMs without any change in the inference complexity.
  arXiv  Detail & Related papers  (2024-03-27T17:50:00Z)
• Discovering modular solutions that generalize compositionally [55.46688816816882]
  We show that identification up to linear transformation purely from demonstrations is possible without having to learn an exponential number of module combinations. We further demonstrate empirically that meta-learning from finite data can discover modular policies that generalize compositionally in a number of complex environments.
  arXiv  Detail & Related papers  (2023-12-22T16:33:50Z)
• ModuleFormer: Modularity Emerges from Mixture-of-Experts [60.6148988099284]
  This paper proposes a new neural network architecture, ModuleFormer, to improve the efficiency and flexibility of large language models. Unlike the previous SMoE-based modular language model, ModuleFormer can induce modularity from uncurated data.
  arXiv  Detail & Related papers  (2023-06-07T17:59:57Z)
• Modular Deep Learning [120.36599591042908]
  Transfer learning has recently become the dominant paradigm of machine learning. It remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference. Modular deep learning has emerged as a promising solution to these challenges.
  arXiv  Detail & Related papers  (2023-02-22T18:11:25Z)
• Is a Modular Architecture Enough? [80.32451720642209]
  We provide a thorough assessment of common modular architectures, through the lens of simple and known modular data distributions. We highlight the benefits of modularity and sparsity and reveal insights on the challenges faced while optimizing modular systems.
  arXiv  Detail & Related papers  (2022-06-06T16:12:06Z)
• Robustness modularity in complex networks [1.749935196721634]
  We propose a new measure based on the concept of robustness. robustness modularity is the probability to find trivial partitions when the structure of the network is randomly perturbed. Tests on artificial and real graphs reveal that robustness modularity can be used to assess and compare the strength of the community structure of different networks.
  arXiv  Detail & Related papers  (2021-10-05T19:00:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.