Tropical Expressivity of Neural Networks

• URL: http://arxiv.org/abs/2405.20174v2
• Date: Tue, 08 Oct 2024 20:58:46 GMT
• Title: Tropical Expressivity of Neural Networks
• Authors: Paul Lezeau, Thomas Walker, Yueqi Cao, Shiv Bhatia, Anthea Monod,
• Abstract summary: We use tropical geometry to characterize and study various architectural aspects of neural networks. We present a new algorithm that computes the exact number of their linear regions.
• Score: 0.0
• License:
• Abstract: We propose an algebraic geometric framework to study the expressivity of linear activation neural networks. A particular quantity of neural networks that has been actively studied is the number of linear regions, which gives a quantification of the information capacity of the architecture. To study and evaluate information capacity and expressivity, we work in the setting of tropical geometry - a combinatorial and polyhedral variant of algebraic geometry - where there are known connections between tropical rational maps and feedforward neural networks. Our work builds on and expands this connection to capitalize on the rich theory of tropical geometry to characterize and study various architectural aspects of neural networks. Our contributions are threefold: we provide a novel tropical geometric approach to selecting sampling domains among linear regions; an algebraic result allowing for a guided restriction of the sampling domain for network architectures with symmetries; and a new open source OSCAR library to analyze neural networks symbolically using their tropical representations, where we present a new algorithm that computes the exact number of their linear regions. We provide a comprehensive set of proof-of-concept numerical experiments demonstrating the breadth of neural network architectures to which tropical geometric theory can be applied to reveal insights on expressivity characteristics of a network. Our work provides the foundations for the adaptation of both theory and existing software from computational tropical geometry and symbolic computation to neural networks and deep learning

Related papers

• Activation thresholds and expressiveness of polynomial neural networks [0.0]
  Polynomial neural networks have been implemented in a range of applications. In this work, we introduce the notion of the activation threshold of a network architecture.
  arXiv  Detail & Related papers  (2024-08-08T16:28:56Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• Defining Neural Network Architecture through Polytope Structures of Dataset [53.512432492636236]
  This paper defines upper and lower bounds for neural network widths, which are informed by the polytope structure of the dataset in question. We develop an algorithm to investigate a converse situation where the polytope structure of a dataset can be inferred from its corresponding trained neural networks. It is established that popular datasets such as MNIST, Fashion-MNIST, and CIFAR10 can be efficiently encapsulated using no more than two polytopes with a small number of faces.
  arXiv  Detail & Related papers  (2024-02-04T08:57:42Z)
• Geometry of Polynomial Neural Networks [3.498371632913735]
  We study the expressivity and learning process for neural networks (PNNs) with monomial activation functions. These theoretical results are accompanied by experiments.
  arXiv  Detail & Related papers  (2024-02-01T19:06:06Z)
• Riemannian Residual Neural Networks [58.925132597945634]
  We show how to extend the residual neural network (ResNet) ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and easy-to-incorporate nature when building varied neural networks.
  arXiv  Detail & Related papers  (2023-10-16T02:12:32Z)
• Deep neural networks architectures from the perspective of manifold learning [0.0]
  This paper is a comprehensive comparison and description of neural network architectures in terms of ge-ometry and topology. We focus on the internal representation of neural networks and on the dynamics of changes in the topology and geometry of a data manifold on different layers.
  arXiv  Detail & Related papers  (2023-06-06T04:57:39Z)
• Quiver neural networks [5.076419064097734]
  We develop a uniform theoretical approach towards the analysis of various neural network connectivity architectures. Inspired by quiver representation theory in mathematics, this approach gives a compact way to capture elaborate data flows.
  arXiv  Detail & Related papers  (2022-07-26T09:42:45Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Neural population geometry: An approach for understanding biological and artificial neural networks [3.4809730725241605]
  We review examples of geometrical approaches providing insight into the function of biological and artificial neural networks. Neural population geometry has the potential to unify our understanding of structure and function in biological and artificial neural networks.
  arXiv  Detail & Related papers  (2021-04-14T18:10:34Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)
• On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspective [54.1171355815052]
  This work tackles the problem of characterizing and understanding the decision boundaries of neural networks with piecewise linear non-linearity activations. We use tropical geometry, a new development in the area of algebraic geometry, to characterize the decision boundaries of a simple network of the form (Affine, ReLU, Affine)
  arXiv  Detail & Related papers  (2020-02-20T16:22:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.