Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

• URL: http://arxiv.org/abs/2607.05546v1
• Date: Mon, 06 Jul 2026 18:29:35 GMT
• Title: Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity
• Abstract summary: We develop a unified function space theory of deep fully connected neural networks.<n>Our framework provides a meaningful notion of functional complexity for deep networks.<n>We show that deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction.
• Score: 19.53343888629072
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We develop a unified function space theory of deep fully connected neural networks. Functions in our spaces are defined recursively as $\ell^1$-bounded linear combinations of activated functions from preceding layers, with a dictionary of affine functions at the first layer. Unlike existing theories that are largely specialized to homogeneous activations such as the ReLU, our framework provides a meaningful notion of functional complexity for deep networks with a broad range of homogeneous and non-homogeneous activation functions commonly used in practice. This simple construction unites several seemingly disparate ideas from the literature, including norm-based complexity bounds and variational characterizations of depth, and facilitates novel analyses of what kinds of functions deep norm-constrained networks can represent. To this end, we prove a novel representer theorem for our spaces and establish novel function-space complexity bounds showing that the associated function classes remain qualitatively small at arbitrary depth. In the univariate ReLU case, we prove a "depth saturation" result: depth in this setting yields only a small constant rescaling of the function class, with no added functional diversity. As a consequence, we show that deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction. This finding reveals that some commonly cited expressivity benefits of depth disappear once network complexity is controlled by an appropriate function space norm, rather than parameter count or other representational costs that permit compounded rescaling across layers. Overall, our results illustrate how a function space perspective yields new structural insights into the relationship between depth and complexity.

Related papers

• GRIFDIR: Graph Resolution-Invariant FEM Diffusion Models in Function Spaces over Irregular Domains [56.121725064621295]
  Score-based diffusion models in function spaces offer a principled framework for modelling function-valued data.<n>However, practical implementations have struggled to fully realise these benefits.<n>We propose a novel architecture that represents generalised convolutional kernels as finite graph kernels.
  arXiv  Detail & Related papers  (2026-05-05T08:33:52Z)
• Function Forms of Simple ReLU Networks with Random Hidden Weights [1.2289361708127877]
  We investigate the function space dynamics of a two-layer ReLU neural network in the infinite-width limit.<n>We highlight the Fisher information matrix's role in steering learning.<n>This work offers a robust foundation for understanding wide neural networks.
  arXiv  Detail & Related papers  (2025-05-23T13:53:02Z)
• Structure of universal formulas [13.794391803767617]
  We introduce a hierarchy of classes connecting the global approximability property to the weaker property of infinite VC dimension. We show that fixed-size neural networks with not more than one layer of neurons having activations cannot approximate functions on arbitrary finite sets. We give examples of functional families, including two-hidden-layer neural networks, that approximate functions on arbitrary finite sets, but fail to do that on the whole domain of definition.
  arXiv  Detail & Related papers  (2023-11-07T11:50:25Z)
• Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory [64.06519549649495]
  We provide the definition of what we call functionally equivalent features. These features produce equivalent output under certain transformations. We propose an efficient algorithm named Iterative Feature Merging.
  arXiv  Detail & Related papers  (2023-10-10T16:27:12Z)
• Data Topology-Dependent Upper Bounds of Neural Network Widths [52.58441144171022]
  We first show that a three-layer neural network can be designed to approximate an indicator function over a compact set. This is then extended to a simplicial complex, deriving width upper bounds based on its topological structure. We prove the universal approximation property of three-layer ReLU networks using our topological approach.
  arXiv  Detail & Related papers  (2023-05-25T14:17:15Z)
• Bayes Complexity of Learners vs Overfitting [4.873362301533825]
  We show that a new notion of complexity of functions governs a PAC Bayes-like generalization bound. In contrast to previous works, our notion naturally generalizes to neural networks with several layers. An upper-bound we derive allows to show a separation in the number of samples needed for good generalization between 2 and 4-layer neural networks.
  arXiv  Detail & Related papers  (2023-03-13T13:07:02Z)
• Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions [63.21838830509772]
  We develop a framework for extending set functions onto low-dimensional continuous domains. Our framework subsumes many well-known extensions as special cases. We convert low-dimensional neural network bottlenecks into representations in high-dimensional spaces.
  arXiv  Detail & Related papers  (2022-08-08T10:58:02Z)
• On the Expected Complexity of Maxout Networks [0.0]
  Recent works have shown that the practical complexity of deep ReLU networks is often far from the theoretical maximum. In this work we show that this phenomenon also occurs in networks with maxout (multi-argument) activation functions. We also show that the parameter space has a multitude of full-dimensional regions with widely different complexity, and obtain nontrivial lower bounds on the expected complexity.
  arXiv  Detail & Related papers  (2021-07-01T11:36:32Z)
• Towards Lower Bounds on the Depth of ReLU Neural Networks [7.355977594790584]
  We investigate whether the class of exactly representable functions strictly increases by adding more layers. We settle an old conjecture about piecewise linear functions by Wang and Sun (2005) in the affirmative. We present upper bounds on the sizes of neural networks required to represent functions with logarithmic depth.
  arXiv  Detail & Related papers  (2021-05-31T09:49:14Z)
• On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spaces [208.67848059021915]
  We study the exploration-exploitation tradeoff at the core of reinforcement learning. In particular, we prove that the complexity of the function class $mathcalF$ characterizes the complexity of the function. Our regret bounds are independent of the number of episodes.
  arXiv  Detail & Related papers  (2020-11-09T18:32:22Z)
• A Functional Perspective on Learning Symmetric Functions with Neural Networks [48.80300074254758]
  We study the learning and representation of neural networks defined on measures. We establish approximation and generalization bounds under different choices of regularization. The resulting models can be learned efficiently and enjoy generalization guarantees that extend across input sizes.
  arXiv  Detail & Related papers  (2020-08-16T16:34:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.