On the Architectural Complexity of Neural Networks

• URL: http://arxiv.org/abs/2605.04325v1
• Date: Tue, 05 May 2026 22:05:34 GMT
• Title: On the Architectural Complexity of Neural Networks
• Abstract summary: We introduce a unified theoretical framework for the rigorous analysis and systematic construction of deep neural networks (DNNs)<n>This framework addresses a gap in existing theory by explicitly modeling the structure of tensor operations.<n>Our study of DNNs introduced over the past 40 years reveals a connection between groundbreaking architectures and increases in different types of architectural complexity.
• Score: 12.289403088028935
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a unified theoretical framework for the rigorous analysis and systematic construction of deep neural networks (DNNs). This framework addresses a gap in existing theory by explicitly modeling the structure of tensor operations -- lower level information that is often abstracted. Our framework enables two novel objectives: (1) analysis of the evolution of architectural complexity over deep learning history, and (2) automatic construction of novel architectures based on new types of tensor operations. Our study of DNNs introduced over the past 40 years reveals a connection between groundbreaking architectures and increases in different types of architectural complexity. Moreover, we identify several large classes of higher complexity architectures that have not yet been explored. We then collect a dataset of 3,000+ higher complexity architectures, which we publicly release at: https://github.com/combinatoriallabs/ArchitecturalComplexity.

Related papers

• BioArc: Discovering Optimal Neural Architectures for Biological Foundation Models [31.218090448573776]
  Foundation models have revolutionized various fields such as natural language processing (NLP) and computer vision (CV)<n>We introduce BioArc, a novel framework designed to move beyond intuition-driven architecture design towards principled, automated architecture discovery for biological foundation models.<n>Our work provides a foundational resource and a principled methodology to guide the creation of the next generation of task-specific and foundation models for biology.
  arXiv  Detail & Related papers  (2025-11-29T02:36:54Z)
• Copresheaf Topological Neural Networks: A Generalized Deep Learning Framework [16.903981913294103]
  We introduce copresheaf topological neural networks (CTNNs)<n>CTNNs are a powerful unifying framework that encapsulates a wide spectrum of deep learning architectures.<n>We show that CTNNs consistently outperform conventional baselines in tasks requiring hierarchical or localized sensitivity.
  arXiv  Detail & Related papers  (2025-05-27T14:28:50Z)
• Unsupervised Graph Neural Architecture Search with Disentangled Self-supervision [51.88848982611515]
  Unsupervised graph neural architecture search remains unexplored in the literature. We propose a novel Disentangled Self-supervised Graph Neural Architecture Search model. Our model is able to achieve state-of-the-art performance against several baseline methods in an unsupervised manner.
  arXiv  Detail & Related papers  (2024-03-08T05:23:55Z)
• Defining Neural Network Architecture through Polytope Structures of Dataset [53.512432492636236]
  This paper defines upper and lower bounds for neural network widths, which are informed by the polytope structure of the dataset in question. We develop an algorithm to investigate a converse situation where the polytope structure of a dataset can be inferred from its corresponding trained neural networks. It is established that popular datasets such as MNIST, Fashion-MNIST, and CIFAR10 can be efficiently encapsulated using no more than two polytopes with a small number of faces.
  arXiv  Detail & Related papers  (2024-02-04T08:57:42Z)
• Serving Deep Learning Model in Relational Databases [70.53282490832189]
  Serving deep learning (DL) models on relational data has become a critical requirement across diverse commercial and scientific domains. We highlight three pivotal paradigms: The state-of-the-art DL-centric architecture offloads DL computations to dedicated DL frameworks. The potential UDF-centric architecture encapsulates one or more tensor computations into User Defined Functions (UDFs) within the relational database management system (RDBMS)
  arXiv  Detail & Related papers  (2023-10-07T06:01:35Z)
• Neural Architecture Retrieval [27.063268631346713]
  We define a new problem Neural Architecture Retrieval which retrieves a set of existing neural architectures with similar designs to the query neural architecture. Existing graph pre-training strategies cannot address the computational graph in neural architectures due to the graph size and motifs. We introduce multi-level contrastive learning to achieve accurate graph representation learning.
  arXiv  Detail & Related papers  (2023-07-16T01:56:41Z)
• Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles [22.66053583920441]
  ArchExplorer is a visual analysis method for understanding a neural architecture space and summarizing design principles. A circle-packing-based architecture visualization has been developed to convey both the global relationships between clusters and local neighborhoods of the architectures in each cluster. Two case studies and a post-analysis are presented to demonstrate the effectiveness of ArchExplorer in summarizing design principles and selecting better-performing architectures.
  arXiv  Detail & Related papers  (2022-08-20T12:15:59Z)
• Redefining Neural Architecture Search of Heterogeneous Multi-Network Models by Characterizing Variation Operators and Model Components [71.03032589756434]
  We investigate the effect of different variation operators in a complex domain, that of multi-network heterogeneous neural models. We characterize both the variation operators, according to their effect on the complexity and performance of the model; and the models, relying on diverse metrics which estimate the quality of the different parts composing it.
  arXiv  Detail & Related papers  (2021-06-16T17:12:26Z)
• Understanding Deep Architectures with Reasoning Layer [60.90906477693774]
  We show that properties of the algorithm layers, such as convergence, stability, and sensitivity, are intimately related to the approximation and generalization abilities of the end-to-end model. Our theory can provide useful guidelines for designing deep architectures with reasoning layers.
  arXiv  Detail & Related papers  (2020-06-24T00:26:35Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.