Block-Operations: Using Modular Routing to Improve Compositional Generalization

• URL: http://arxiv.org/abs/2408.00508v1
• Date: Thu, 1 Aug 2024 12:28:22 GMT
• Title: Block-Operations: Using Modular Routing to Improve Compositional Generalization
• Authors: Florian Dietz, Dietrich Klakow,
• Abstract summary: We introduce the Multiplexer, a new architectural component that enhances the Feed Forward Neural Network (FNN) We experimentally confirm that Multiplexers exhibit strong compositional generalization.
• Score: 17.525220958618988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We explore the hypothesis that poor compositional generalization in neural networks is caused by difficulties with learning effective routing. To solve this problem, we propose the concept of block-operations, which is based on splitting all activation tensors in the network into uniformly sized blocks and using an inductive bias to encourage modular routing and modification of these blocks. Based on this concept we introduce the Multiplexer, a new architectural component that enhances the Feed Forward Neural Network (FNN). We experimentally confirm that Multiplexers exhibit strong compositional generalization. On both a synthetic and a realistic task our model was able to learn the underlying process behind the task, whereas both FNNs and Transformers were only able to learn heuristic approximations. We propose as future work to use the principles of block-operations to improve other existing architectures.

Related papers

• Detecting and Approximating Redundant Computational Blocks in Neural Networks [25.436785396394804]
  intra-network similarities present new opportunities for designing more efficient neural networks. We introduce a simple metric, Block Redundancy, to detect redundant blocks, and propose Redundant Blocks Approximation (RBA) to approximate redundant blocks. RBA reduces model parameters and time complexity while maintaining good performance.
  arXiv  Detail & Related papers  (2024-10-07T11:35:24Z)
• Breaking Neural Network Scaling Laws with Modularity [8.482423139660153]
  We show how the amount of training data required to generalize varies with the intrinsic dimensionality of a task's input. We then develop a novel learning rule for modular networks to exploit this advantage.
  arXiv  Detail & Related papers  (2024-09-09T16:43:09Z)
• Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning [0.0]
  We show that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics. We demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales.
  arXiv  Detail & Related papers  (2024-06-10T13:44:07Z)
• Task Agnostic Architecture for Algorithm Induction via Implicit Composition [10.627575117586417]
  This position paper aims to explore developing such a unified architecture and proposes a theoretical framework of how it could be constructed. Recent Generative AI, especially Transformer-based models, demonstrate potential as an architecture capable of constructing algorithms for a wide range of domains. Our exploration delves into current capabilities and limitations of Transformer-based and other methods in efficient and correct algorithm composition.
  arXiv  Detail & Related papers  (2024-04-03T04:31:09Z)
• Distance Weighted Trans Network for Image Completion [52.318730994423106]
  We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. CNNs are used to augment the local texture information of coarse priors. DWT blocks are used to recover certain coarse textures and coherent visual structures.
  arXiv  Detail & Related papers  (2023-10-11T12:46:11Z)
• Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One [60.5818387068983]
  Graph neural networks (GNN) suffer from severe inefficiency. We propose to decouple a multi-layer GNN as multiple simple modules for more efficient training. We show that the proposed framework is highly efficient with reasonable performance.
  arXiv  Detail & Related papers  (2023-04-20T07:21:32Z)
• Defensive Tensorization [113.96183766922393]
  We propose tensor defensiveization, an adversarial defence technique that leverages a latent high-order factorization of the network. We empirically demonstrate the effectiveness of our approach on standard image classification benchmarks. We validate the versatility of our approach across domains and low-precision architectures by considering an audio task and binary networks.
  arXiv  Detail & Related papers  (2021-10-26T17:00:16Z)
• Reinforcement Learning with External Knowledge by using Logical Neural Networks [67.46162586940905]
  A recent neuro-symbolic framework called the Logical Neural Networks (LNNs) can simultaneously provide key-properties of both neural networks and symbolic logic. We propose an integrated method that enables model-free reinforcement learning from external knowledge sources.
  arXiv  Detail & Related papers  (2021-03-03T12:34:59Z)
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers [84.57980167400513]
  Neural Function Modules (NFM) aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm.
  arXiv  Detail & Related papers  (2020-10-15T20:43:17Z)
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks [10.0444013205203]
  Understanding if and how NNs are modular could provide insights into how to improve them. Current inspection methods, however, fail to link modules to their functionality.
  arXiv  Detail & Related papers  (2020-10-05T15:04:11Z)
• Deep Multi-Task Learning for Cooperative NOMA: System Design and Principles [52.79089414630366]
  We develop a novel deep cooperative NOMA scheme, drawing upon the recent advances in deep learning (DL) We develop a novel hybrid-cascaded deep neural network (DNN) architecture such that the entire system can be optimized in a holistic manner.
  arXiv  Detail & Related papers  (2020-07-27T12:38:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.