Singular Learning and Occam's Razor in Deep Monomial Networks

• URL: http://arxiv.org/abs/2606.28464v1
• Date: Fri, 26 Jun 2026 13:40:14 GMT
• Title: Singular Learning and Occam's Razor in Deep Monomial Networks
• Abstract summary: In the optimization of neural networks, dynamics are influenced by critical points that arise from the model's architecture.<n>We investigate such points in deep fully-connected networks with monomial activations via algebra.<n>This offers a mathematical perspective on the implicit bias in deep neural networks, explaining the tendency of these models to converge toward simpler functions.
• Score: 6.602002946878127
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the optimization of neural networks, gradient dynamics are influenced by critical points that arise from the model's architecture. These critical points occur where the Jacobian of the model's parametrization is rank-deficient, and are the most pronounced singularities studied in Singular Learning Theory. We investigate such points in deep fully-connected networks with monomial activations via tools from polynomial algebra such as Mason's Theorem. We show that, for sufficiently large activation degree, criticality occurs precisely at subnetworks, i.e., at parameter configurations where some neurons are inactive or redundant. This offers a mathematical perspective on the implicit bias in deep neural networks, explaining the tendency of these models to converge toward simpler functions.

Related papers

• Approximation Theory for Neural Networks: Old and New [1.6300219183960216]
  Universal approximations provide a mathematical explanation for the expressive power of neural networks.<n>We review classical density results for single-hidden-layer networks, as well as quantitative bounds that relate approximation error to network size.<n>Recent developments on Kolmogorov--Arnold Networks (KANs) offer an alternative architectural paradigm.
  arXiv  Detail & Related papers  (2026-05-20T17:42:34Z)
• Learning on a Razor's Edge: the Singularity Bias of Polynomial Neural Networks [2.9816332334719773]
  We consider fully-connected networks with activation functions, and focus on the function space they pararize.<n>First, we compute the dimension of the subspace of the neuromanifold parametrized by neuroworks.<n>Second, we show that this subspace is singular.<n>Third, we argue that such singularities often correspond to critical points of the training dynamics.
  arXiv  Detail & Related papers  (2025-05-17T05:11:17Z)
• The Spectral Bias of Shallow Neural Network Learning is Shaped by the Choice of Non-linearity [0.7499722271664144]
  We study how non-linear activation functions contribute to shaping neural networks' implicit bias.<n>We show that local dynamical attractors facilitate the formation of clusters of hyperplanes where the input to a neuron's activation function is zero.
  arXiv  Detail & Related papers  (2025-03-13T17:36:46Z)
• Towards Explaining Hypercomplex Neural Networks [6.543091030789653]
  Hypercomplex neural networks are gaining increasing interest in the deep learning community. In this paper, we propose inherently interpretable PHNNs and quaternion-like networks. We draw insights into how this unique branch of neural models operates.
  arXiv  Detail & Related papers  (2024-03-26T17:58:07Z)
• Riemannian Residual Neural Networks [58.925132597945634]
  We show how to extend the residual neural network (ResNet) ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and easy-to-incorporate nature when building varied neural networks.
  arXiv  Detail & Related papers  (2023-10-16T02:12:32Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth [92.25666446274188]
  sinusoidal neural networks with activations have been proposed as an alternative to networks with traditional activation functions. We first propose a simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis. We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kernel approximates a low-pass filter with an adjustable bandwidth.
  arXiv  Detail & Related papers  (2022-11-26T07:41:48Z)
• Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications [6.579523168465526]
  We introduce emphpartial Jacobians of a network, defined as derivatives of preactivations in layer $l$ with respect to preactivations in layer $l_0leq l$. We derive recurrence relations for the norms of partial Jacobians and utilize these relations to analyze criticality of deep fully connected neural networks with LayerNorm and/or residual connections.
  arXiv  Detail & Related papers  (2021-11-23T20:31:42Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)
• Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
  We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
  arXiv  Detail & Related papers  (2020-02-20T15:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.