Neural Sequence-to-grid Module for Learning Symbolic Rules

• URL: http://arxiv.org/abs/2101.04921v1
• Date: Wed, 13 Jan 2021 07:53:14 GMT
• Title: Neural Sequence-to-grid Module for Learning Symbolic Rules
• Authors: Segwang Kim, Hyoungwook Nam, Joonyoung Kim, Kyomin Jung
• Abstract summary: We propose a neural sequence-to-grid (seq2grid) module, an input preprocessor that automatically segments and aligns an input sequence into a grid. As our module outputs a grid via a novel differentiable mapping, any neural network structure taking a grid input, such as ResNet or TextCNN, can be jointly trained with our module.
• Score: 14.946594806351971
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Logical reasoning tasks over symbols, such as learning arithmetic operations and computer program evaluations, have become challenges to deep learning. In particular, even state-of-the-art neural networks fail to achieve \textit{out-of-distribution} (OOD) generalization of symbolic reasoning tasks, whereas humans can easily extend learned symbolic rules. To resolve this difficulty, we propose a neural sequence-to-grid (seq2grid) module, an input preprocessor that automatically segments and aligns an input sequence into a grid. As our module outputs a grid via a novel differentiable mapping, any neural network structure taking a grid input, such as ResNet or TextCNN, can be jointly trained with our module in an end-to-end fashion. Extensive experiments show that neural networks having our module as an input preprocessor achieve OOD generalization on various arithmetic and algorithmic problems including number sequence prediction problems, algebraic word problems, and computer program evaluation problems while other state-of-the-art sequence transduction models cannot. Moreover, we verify that our module enhances TextCNN to solve the bAbI QA tasks without external memory.

Related papers

• LinSATNet: The Positive Linear Satisfiability Neural Networks [116.65291739666303]
  This paper studies how to introduce the popular positive linear satisfiability to neural networks. We propose the first differentiable satisfiability layer based on an extension of the classic Sinkhorn algorithm for jointly encoding multiple sets of marginal distributions.
  arXiv  Detail & Related papers  (2024-07-18T22:05:21Z)
• The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks [59.26515696183751]
  We show that algorithm discovery in neural networks is sometimes more complex. We show that even simple learning problems can admit a surprising diversity of solutions.
  arXiv  Detail & Related papers  (2023-06-30T17:59:13Z)
• A Neural Lambda Calculus: Neurosymbolic AI meets the foundations of computing and functional programming [0.0]
  We will analyze the ability of neural networks to learn how to execute programs as a whole. We will introduce the use of integrated neural learning and calculi formalization.
  arXiv  Detail & Related papers  (2023-04-18T20:30:16Z)
• Permutation Equivariant Neural Functionals [92.0667671999604]
  This work studies the design of neural networks that can process the weights or gradients of other neural networks. We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-02-27T18:52:38Z)
• Pathfinding Neural Cellular Automata [23.831530224401575]
  Pathfinding is an important sub-component of a broad range of complex AI tasks, such as robot path planning, transport routing, and game playing. We hand-code and learn models for Breadth-First Search (BFS), i.e. shortest path finding. We present a neural implementation of Depth-First Search (DFS), and outline how it can be combined with neural BFS to produce an NCA for computing diameter of a graph. We experiment with architectural modifications inspired by these hand-coded NCAs, training networks from scratch to solve the diameter problem on grid mazes while exhibiting strong ability generalization
  arXiv  Detail & Related papers  (2023-01-17T11:45:51Z)
• A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms [64.3064050603721]
  We generalize Runge-Kutta neural network to a recurrent neural network (R2N2) superstructure for the design of customized iterative algorithms. We demonstrate that regular training of the weight parameters inside the proposed superstructure on input/output data of various computational problem classes yields similar iterations to Krylov solvers for linear equation systems, Newton-Krylov solvers for nonlinear equation systems, and Runge-Kutta solvers for ordinary differential equations.
  arXiv  Detail & Related papers  (2022-11-22T16:30:33Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Modeling Structure with Undirected Neural Networks [20.506232306308977]
  We propose undirected neural networks, a flexible framework for specifying computations that can be performed in any order. We demonstrate the effectiveness of undirected neural architectures, both unstructured and structured, on a range of tasks.
  arXiv  Detail & Related papers  (2022-02-08T10:06:51Z)
• Extending Answer Set Programs with Neural Networks [2.512827436728378]
  We propose NeurASP -- a simple extension of answer set programs by embracing neural networks. We show that NeurASP can not only improve the perception accuracy of a pre-trained neural network, but also help to train a neural network better by giving restrictions through logic rules.
  arXiv  Detail & Related papers  (2020-09-22T00:52:30Z)
• RE-MIMO: Recurrent and Permutation Equivariant Neural MIMO Detection [85.44877328116881]
  We present a novel neural network for symbol detection in wireless communication systems. It is motivated by several important considerations in wireless communication systems. We compare its performance against existing methods and the results show the ability of our network to efficiently handle a variable number of transmitters.
  arXiv  Detail & Related papers  (2020-06-30T22:43:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.