Compiling to recurrent neurons

• URL: http://arxiv.org/abs/2511.14953v1
• Date: Tue, 18 Nov 2025 22:26:27 GMT
• Title: Compiling to recurrent neurons
• Authors: Joey Velez-Ginorio, Nada Amin, Konrad Kording, Steve Zdancewic,
• Abstract summary: We present a minimal typed, higher-order and linear programming language with iteration called $textsfCajalscriptstyle(mathbbmultimap, mathbb2, mathbbN)$.<n>We prove its programs compile correctly to recurrent neurons, allowing discrete algorithms to be expressed in a differentiable form compatible with gradient-based learning.
• Score: 0.5249805590164902
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete structures are currently second-class in differentiable programming. Since functions over discrete structures lack overt derivatives, differentiable programs do not differentiate through them and limit where they can be used. For example, when programming a neural network, conditionals and iteration cannot be used everywhere; they can break the derivatives necessary for gradient-based learning to work. This limits the class of differentiable algorithms we can directly express, imposing restraints on how we build neural networks and differentiable programs more generally. However, these restraints are not fundamental. Recent work shows conditionals can be first-class, by compiling them into differentiable form as linear neurons. Similarly, this work shows iteration can be first-class -- by compiling to linear recurrent neurons. We present a minimal typed, higher-order and linear programming language with iteration called $\textsf{Cajal}\scriptstyle(\mathbb{\multimap}, \mathbb{2}, \mathbb{N})$. We prove its programs compile correctly to recurrent neurons, allowing discrete algorithms to be expressed in a differentiable form compatible with gradient-based learning. With our implementation, we conduct two experiments where we link these recurrent neurons against a neural network solving an iterative image transformation task. This determines part of its function prior to learning. As a result, the network learns faster and with greater data-efficiency relative to a neural network programmed without first-class iteration. A key lesson is that recurrent neurons enable a rich interplay between learning and the discrete structures of ordinary programming.

Related papers

• Compiling to linear neurons [0.5249805590164902]
  We don't program neural networks directly. Instead, we rely on an indirect style where learning algorithms determine a neural network's function by learning from data.<n>We can't compile most algorithms into a neural network -- even if these algorithms could help the network learn.<n>We introduce $textsfCajal$: a typed, higher-order and linear programming language intended to be a minimal vehicle for exploring a direct style of programming neural networks.
  arXiv  Detail & Related papers  (2025-11-14T22:19:50Z)
• Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning [78.88684753303794]
  Deep learning has predominantly advanced through applications in computer vision and natural language processing.<n>Neural operators are a principled way to generalize neural networks to mappings between function spaces.<n>This paper identifies and distills the key principles for constructing practical implementations of mappings between infinite-dimensional function spaces.
  arXiv  Detail & Related papers  (2025-06-12T17:59:31Z)
• LinSATNet: The Positive Linear Satisfiability Neural Networks [116.65291739666303]
  This paper studies how to introduce the popular positive linear satisfiability to neural networks. We propose the first differentiable satisfiability layer based on an extension of the classic Sinkhorn algorithm for jointly encoding multiple sets of marginal distributions.
  arXiv  Detail & Related papers  (2024-07-18T22:05:21Z)
• Taming Binarized Neural Networks and Mixed-Integer Programs [2.7624021966289596]
  We show that binarized neural networks admit a tame representation. This makes it possible to use the framework of Bolte et al. for implicit differentiation. This approach could also be used for a broader class of mixed-integer programs.
  arXiv  Detail & Related papers  (2023-10-05T21:04:16Z)
• Efficient Vectorized Backpropagation Algorithms for Training Feedforward Networks Composed of Quadratic Neurons [1.474723404975345]
  This paper presents a solution to the XOR problem with a single quadratic neuron.<n>It shows that any dataset composed of $mathcalC$ bounded clusters can be separated with only a single layer of $mathcalC$ quadratic neurons.
  arXiv  Detail & Related papers  (2023-10-04T15:39:57Z)
• The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks [59.26515696183751]
  We show that algorithm discovery in neural networks is sometimes more complex. We show that even simple learning problems can admit a surprising diversity of solutions.
  arXiv  Detail & Related papers  (2023-06-30T17:59:13Z)
• Towards a Neural Lambda Calculus: Neurosymbolic AI Applied to the Foundations of Functional Programming [0.0]
  We will analyze the ability of neural networks to learn how to execute programs as a whole.<n>We will introduce the use of integrated neural learning and calculi formalization.
  arXiv  Detail & Related papers  (2023-04-18T20:30:16Z)
• A Tutorial on Neural Networks and Gradient-free Training [0.0]
  This paper presents a compact, matrix-based representation of neural networks in a self-contained tutorial fashion. neural networks are mathematical nonlinear functions constructed by composing several vector-valued functions.
  arXiv  Detail & Related papers  (2022-11-26T15:33:11Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations [160.33479656108926]
  In this work, we demonstrate that intermediate neural representations add more flexibility to neural networks. We show that neural representation can achieve improved sample complexities compared with the raw input. Our results characterize when neural representations are beneficial, and may provide a new perspective on why depth is important in deep learning.
  arXiv  Detail & Related papers  (2020-06-24T02:44:54Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)
• Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning [66.05472746340142]
  This paper analyzes how multi-layer neural networks can perform hierarchical learning _efficiently_ and _automatically_ by SGD on the training objective. We establish a new principle called "backward feature correction", where the errors in the lower-level features can be automatically corrected when training together with the higher-level layers.
  arXiv  Detail & Related papers  (2020-01-13T17:28:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.