Low-dimensional topology of deep neural networks

• URL: http://arxiv.org/abs/2606.31856v1
• Date: Tue, 30 Jun 2026 15:53:11 GMT
• Title: Low-dimensional topology of deep neural networks
• Abstract summary: We study layered models, including feedforward networks, ResNets, and transformers, by limiting each layer to a width of $d = 3$.<n>This allows us to track how a neural network changes low-dimensional topological invariants through its layers.
• Score: 2.396810202210905
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study layered models, including feedforward networks, ResNets, and transformers, by limiting each layer to a width of $d = 3$, i.e., $\mathbb{R}^3$ as representation space. This allows us to track how a neural network changes low-dimensional topological invariants through its layers. Just about any topological structure may be simplified or even trivialized by simply increasing dimension; e.g., any knot is equivalent to an unknot in $\mathbb{R}^4$. By restricting to $\mathbb{R}^3$, we not only isolate the effects of activation and depth from that of width, we work in a space that lends itself to easy visualization. We focus on linking number here, deferring other invariants like link groups, Milnor's $\barμ$-invariants, knot types, ambient cobordisms, to a sequel. We provide full proofs and empirical experiments to justify the following insights: When measured by their power to effect changes in linking numbers, the layer-skipping feature in ResNets is as powerful as the attention mechanism in transformers; both ResNets and transformers are strictly more powerful than feedforward neural networks with monotonic activations, which are in turn more powerful than invertible and flow-based models; but replacing monotonic activation with a nonmonotonic one elevates a feedforward network into the same expressivity class as ResNets and transformers. These results suggest that low-dimensional topology can be a useful tool to guide designs of AI architectures. We also generalize our results from $d = 3$ to arbitrary $d > 3$.

Related papers

• NN-Former: Rethinking Graph Structure in Neural Architecture Representation [67.3378579108611]
  Graph Neural Networks (GNNs) and transformers have shown promising performance in representing neural architectures.<n>We show that sibling nodes are pivotal while overlooked in previous research.<n>Our approach consistently achieves promising performance in both accuracy and latency prediction.
  arXiv  Detail & Related papers  (2025-07-01T15:46:18Z)
• On the Role of Depth and Looping for In-Context Learning with Task Diversity [69.4145579827826]
  We study in-context learning for linear regression with diverse tasks. We show that multilayer Transformers are not robust to even distributional shifts as small as $O(e-L)$ in Wasserstein distance.
  arXiv  Detail & Related papers  (2024-10-29T03:27:56Z)
• Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets [58.460298576330835]
  We study Leaky ResNets, which interpolate between ResNets and Fully-Connected nets depending on an 'effective depth'<n>We leverage this intuition to explain the emergence of a bottleneck structure, as observed in previous work.
  arXiv  Detail & Related papers  (2024-05-27T18:15:05Z)
• The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes [75.59720049837459]
  We study the transition from infinite-width behavior to this variance limited regime as a function of sample size $P$ and network width $N$. We find that finite-size effects can become relevant for very small datasets on the order of $P* sim sqrtN$ for regression with ReLU networks.
  arXiv  Detail & Related papers  (2022-12-23T04:48:04Z)
• A Feedforward Unitary Equivariant Neural Network [3.6220250022337335]
  We devise a new type of feedforward neural network. It is equivariant with respect to the unitary group $U(n)$. The input and output can be vectors in $mathbbCn$ with arbitrary dimension $n$.
  arXiv  Detail & Related papers  (2022-08-25T15:05:02Z)
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics [8.160343645537106]
  We develop Banach spaces for ReLU neural networks of finite depth $L$ and infinite width. The spaces contain all finite fully connected $L$-layer networks and their $L2$-limiting objects under on the natural path-norm. Under this norm, the unit ball in the space for $L$-layer networks has low Rademacher complexity and thus favorable properties.
  arXiv  Detail & Related papers  (2020-07-30T17:47:05Z)
• Deep Polynomial Neural Networks [77.70761658507507]
  $Pi$Nets are a new class of function approximators based on expansions. $Pi$Nets produce state-the-art results in three challenging tasks, i.e. image generation, face verification and 3D mesh representation learning.
  arXiv  Detail & Related papers  (2020-06-20T16:23:32Z)
• Topology of deep neural networks [8.946655323517092]
  We study how the topology of a data set $M = M_a cup M_b subseteq mathbbRd$ changes as it passes through the layers of a well-trained neural network.
  arXiv  Detail & Related papers  (2020-04-13T17:53:36Z)
• $\Pi-$nets: Deep Polynomial Neural Networks [86.36557534288535]
  $Pi$-Nets are neural networks in which the output is a high-order of the input. We empirically demonstrate that $Pi$-Nets have better representation power than standard DCNNs. Our framework elucidates why recent generative models, such as StyleGAN, improve upon their predecessors.
  arXiv  Detail & Related papers  (2020-03-08T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.