Functorial Neural Architectures from Higher Inductive Types

• URL: http://arxiv.org/abs/2603.16123v1
• Date: Tue, 17 Mar 2026 05:08:30 GMT
• Title: Functorial Neural Architectures from Higher Inductive Types
• Abstract summary: We show that neural networks systematically fail at compositional generalization.<n>This failure is architectural: compositional generalization is equivalent to functoriality of the decoder.<n>We prove that decoders assembled by structural concatenation of independently generated segments are strict monoidal functors.
• Score: 5.076419064097734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks systematically fail at compositional generalization -- producing correct outputs for novel combinations of known parts. We show that this failure is architectural: compositional generalization is equivalent to functoriality of the decoder, and this perspective yields both guarantees and impossibility results. We compile Higher Inductive Type (HIT) specifications into neural architectures via a monoidal functor from the path groupoid of a target space to a category of parametric maps: path constructors become generator networks, composition becomes structural concatenation, and 2-cells witnessing group relations become learned natural transformations. We prove that decoders assembled by structural concatenation of independently generated segments are strict monoidal functors (compositional by construction), while softmax self-attention is not functorial for any non-trivial compositional task. Both results are formalized in Cubical Agda. Experiments on three spaces validate the full hierarchy: on the torus ($\mathbb{Z}^2$), functorial decoders outperform non-functorial ones by 2-2.7x; on $S^1 \vee S^1$ ($F_2$), the type-A/B gap widens to 5.5-10x; on the Klein bottle ($\mathbb{Z} \rtimes \mathbb{Z}$), a learned 2-cell closes a 46% error gap on words exercising the group relation.

Related papers

• Algebraic Operator Decomposition: A Partitioned Architecture for Noise-Resilient Quantum Computing [0.0]
  We present an operator-decomposition architecture that mathematically maps a global operator into independently executable local operators.<n>AOD complements quantum error correction and error-mitigation approaches by performing algebraic decomposition before quantum execution.
  arXiv  Detail & Related papers  (2026-09-03T16:44:39Z)
• Operator-Theoretic Generalization Bounds for Multitask Deep Learning [0.0254890465057467]
  We develop operator-theoretic generalization bounds for deep multi-output function classes.<n>We derive Rademacher bounds complexity for invertible and width-expanding injective architectures.
  arXiv  Detail & Related papers  (2026-08-17T00:36:57Z)
• Spectral Edge Dynamics Reveal Functional Modes of Learning [0.0]
  Training dynamics during grokking concentrate along a small number of dominant update directions -- the spectral edge.<n>We show that standard mechanistic interpretability tools fail to capture these directions.<n>For modular addition, all leading directions collapse to a single Fourier mode.<n>For multiplication, the same collapse appears only in the discrete-log basis, yielding a 5.9x improvement in concentration.
  arXiv  Detail & Related papers  (2026-04-06T22:29:00Z)
• RiboSphere: Learning Unified and Efficient Representations of RNA Structures [57.40815107640066]
  RNA backbones are highly flexible, non-canonical interactions are prevalent, and experimentally determined 3D structures are comparatively scarce.<n>We introduce emphRiboSphere, a framework that learns emphdiscrete geometric representations of RNA by combining vector quantization with flow matching.
  arXiv  Detail & Related papers  (2026-03-20T04:36:35Z)
• Classical and Quantum Algorithms for Topological Invariants of Torus Bundles [0.42970700836450476]
  We exploit the non-commutative torus structure to embed the skein algebra of the closed torus into its symmetric subalgebra at roots of unity.<n>We prove that extracting individual expansion coefficients is #P-complete, yet there is a quantum algorithm that can efficiently approximate these coefficients for a non-negligible fraction of configurations.
  arXiv  Detail & Related papers  (2025-12-22T04:58:14Z)
• A Kernel-based Stochastic Approximation Framework for Nonlinear Operator Learning [7.820614736576814]
  We develop a framework for approximations between infinite-dimensional spaces using general Mercer operator-valued kernels.<n>Within this framework, we establish dimension-free convergence rates, demonstrating that nonlinear operator learning can overcome the curse of dimensionality.<n>This framework accommodates a wide range of operator learning tasks, ranging from integral operators to architectures based on encoder-decoder representations.
  arXiv  Detail & Related papers  (2025-09-14T03:33:36Z)
• Parseval Convolution Operators and Neural Networks [16.78532039510369]
  We first identify the Parseval convolution operators as the class of energy-preserving filterbanks. We then present a constructive approach for the design/specification of such filterbanks via the chaining of elementary Parseval modules. We demonstrate the usage of those tools with the design of a CNN-based algorithm for the iterative reconstruction of biomedical images.
  arXiv  Detail & Related papers  (2024-08-19T13:31:16Z)
• Scalable Graph Compressed Convolutions [68.85227170390864]
  We propose a differentiable method that applies permutations to calibrate input graphs for Euclidean convolution. Based on the graph calibration, we propose the Compressed Convolution Network (CoCN) for hierarchical graph representation learning.
  arXiv  Detail & Related papers  (2024-07-26T03:14:13Z)
• Learning Polynomial Problems with $SL(2,\mathbb{R})$ Equivariance [6.5783892500847205]
  We show that neural networks can effectively solve problems in a data-driven fashion, achieving tenfold speedups while retaining high accuracy. We observe that these learning problems are equivariant to the non-compact group $SL(2,mathbbR)$, which consists of area-preserving linear transformations.
  arXiv  Detail & Related papers  (2023-12-04T18:59:19Z)
• Factorizers for Distributed Sparse Block Codes [45.29870215671697]
  We propose a fast and highly accurate method for factorizing distributed block codes (SBCs) Our iterative factorizer introduces a threshold-based nonlinear activation, conditional random sampling, and an $ell_infty$-based similarity metric. We demonstrate the feasibility of our method on four deep CNN architectures over CIFAR-100, ImageNet-1K, and RAVEN datasets.
  arXiv  Detail & Related papers  (2023-03-24T12:31:48Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• Equivariant Transduction through Invariant Alignment [71.45263447328374]
  We introduce a novel group-equivariant architecture that incorporates a group-in hard alignment mechanism. We find that our network's structure allows it to develop stronger equivariant properties than existing group-equivariant approaches. We additionally find that it outperforms previous group-equivariant networks empirically on the SCAN task.
  arXiv  Detail & Related papers  (2022-09-22T11:19:45Z)
• Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks [86.10875837475783]
  Systematic compositionality is an essential mechanism in human language, allowing the recombination of known parts to create novel expressions. Existing neural models have been shown to lack this basic ability in learning symbolic structures. We propose two auxiliary sequence prediction tasks that track the progress of function and argument semantics.
  arXiv  Detail & Related papers  (2021-09-30T16:41:19Z)
• Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design [70.27706384570723]
  We propose Fold2Seq, a novel framework for designing protein sequences conditioned on a specific target fold. We show improved or comparable performance of Fold2Seq in terms of speed, coverage, and reliability for sequence design. The unique advantages of fold-based Fold2Seq, in comparison to a structure-based deep model and RosettaDesign, become more evident on three additional real-world challenges.
  arXiv  Detail & Related papers  (2021-06-24T14:34:24Z)
• Abelian Neural Networks [48.52497085313911]
  We first construct a neural network architecture for Abelian group operations and derive a universal approximation property. We extend it to Abelian semigroup operations using the characterization of associative symmetrics. We train our models over fixed word embeddings and demonstrate improved performance over the original word2vec.
  arXiv  Detail & Related papers  (2021-02-24T11:52:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.