Compositional Sparsity as an Inductive Bias for Neural Architecture Design

• URL: http://arxiv.org/abs/2605.14764v1
• Date: Thu, 14 May 2026 12:26:50 GMT
• Title: Compositional Sparsity as an Inductive Bias for Neural Architecture Design
• Abstract summary: We identify the structural priors that enable Deep Neural Networks to overcome the curse of dimensionality.<n>We present an interpretable pipeline in which abstraction emerges through hierarchical composition.<n>Across a broad suite of real-world datasets, HNNs consistently match or outperform dense baselines while using far fewer parameters.
• Score: 35.06894725394093
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Identifying the structural priors that enable Deep Neural Networks (DNNs) to overcome the curse of dimensionality is a fundamental challenge in machine learning theory. Existing literature suggests that effective high-dimensional learning is driven by compositional sparsity, where target functions decompose into constituents supported on low-dimensional variable subsets. To investigate this hypothesis, we combine Information Filtering Networks (IFNs), which extract sparse dependency structures via constrained information maximisation, with Homological Neural Networks (HNNs), which map the inferred topology into fixed-wiring sparse neural graphs. We formalise the design principles underlying this construction and present an interpretable pipeline in which abstraction emerges through hierarchical composition. HNNs are orders of magnitude sparser than standard DNNs and require only minimal hyperparameter tuning. On synthetic tasks with known sparse hierarchies, HNNs recover the underlying compositional structure and remain stable in regimes where dense alternatives degrade as dimensionality increases. Across a broad suite of real-world datasets, HNNs consistently match or outperform dense baselines while using far fewer parameters, exhibiting lower variance and showing reduced sensitivity to hyperparameters.

Related papers

• Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions [14.326511406760822]
  Deep neural networks (DNNs) have achieved remarkable success on complex machine-learning tasks.<n>We show that sublinearly structured DNNs can attain feature-learning consistency comparable to that of classical models.<n>A structural audit indicates that widely used convolutional neural networks (CNNs), including AlexNet, VGGNet, ResNet, GoogLeNet, are sublinearly structured on their image classification benchmarks.
  arXiv  Detail & Related papers  (2026-06-22T15:24:53Z)
• Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime [24.053364183688874]
  Training dynamics is central to understanding neural networks.<n>We propose a convergence framework for analyzing gradient descent dynamics under a broad family of neural network architectures.
  arXiv  Detail & Related papers  (2026-06-22T14:00:26Z)
• Mathematical Modeling and Convergence Analysis of Deep Neural Networks with Dense Layer Connectivities in Deep Learning [1.5516092077598485]
  In deep learning, dense layer connectivity has become a key design principle in deep neural networks (DNNs)<n>In this work, we model densely connected DNNs mathematically and analyze their learning problems in the deep-layer limit.
  arXiv  Detail & Related papers  (2025-10-02T14:22:51Z)
• Spatiotemporal Graph Learning with Direct Volumetric Information Passing and Feature Enhancement [62.91536661584656]
  We propose a dual-module framework, Cell-embedded and Feature-enhanced Graph Neural Network (aka, CeFeGNN) for learning.<n>We embed learnable cell attributions to the common node-edge message passing process, which better captures the spatial dependency of regional features.<n>Experiments on various PDE systems and one real-world dataset demonstrate that CeFeGNN achieves superior performance compared with other baselines.
  arXiv  Detail & Related papers  (2024-09-26T16:22:08Z)
• On Characterizing the Evolution of Embedding Space of Neural Networks using Algebraic Topology [9.537910170141467]
  We study how the topology of feature embedding space changes as it passes through the layers of a well-trained deep neural network (DNN) through Betti numbers. We demonstrate that as depth increases, a topologically complicated dataset is transformed into a simple one, resulting in Betti numbers attaining their lowest possible value.
  arXiv  Detail & Related papers  (2023-11-08T10:45:12Z)
• MuseGNN: Forming Scalable, Convergent GNN Layers that Minimize a Sampling-Based Energy [27.328628715885287]
  A graph neural network (GNN) architecture is capable of modeling data with cross-instance relations.<n>An important subclass of this architecture involves layers designed such that the forward pass iteratively reduces a graph-regularized energy function of interest.<n>We propose a sampling-based energy function and scalable GNN layers that iteratively reduce it, guided by convergence guarantees in certain settings.<n>We also instantiate a full GNN architecture based on these designs, and the model achieves competitive accuracy and scalability when applied to the largest publicly-available node classification benchmark exceeding 1TB in size.
  arXiv  Detail & Related papers  (2023-10-19T04:30:14Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• On the Intrinsic Structures of Spiking Neural Networks [66.57589494713515]
  Recent years have emerged a surge of interest in SNNs owing to their remarkable potential to handle time-dependent and event-driven data. There has been a dearth of comprehensive studies examining the impact of intrinsic structures within spiking computations. This work delves deep into the intrinsic structures of SNNs, by elucidating their influence on the expressivity of SNNs.
  arXiv  Detail & Related papers  (2022-06-21T09:42:30Z)
• STN: Scalable Tensorizing Networks via Structure-Aware Training and Adaptive Compression [10.067082377396586]
  We propose Scalableizing Networks (STN), which adaptively adjust the model size and decomposition structure without retraining. STN is compatible with arbitrary network architectures and achieves higher compression performance and flexibility over other tensorizing versions.
  arXiv  Detail & Related papers  (2022-05-30T15:50:48Z)
• Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis [94.64007376939735]
  We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training. We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
  arXiv  Detail & Related papers  (2022-05-11T17:43:54Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.