Dynamical systems' based neural networks

• URL: http://arxiv.org/abs/2210.02373v2
• Date: Thu, 31 Aug 2023 17:12:16 GMT
• Title: Dynamical systems' based neural networks
• Authors: Elena Celledoni, Davide Murari, Brynjulf Owren, Carola-Bibiane Sch\"onlieb, Ferdia Sherry
• Abstract summary: We build neural networks using a suitable, structure-preserving, numerical time-discretisation. The structure of the neural network is then inferred from the properties of the ODE vector field. We present two universal approximation results and demonstrate how to impose some particular properties on the neural networks.
• Score: 0.7874708385247353
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks have gained much interest because of their effectiveness in many applications. However, their mathematical properties are generally not well understood. If there is some underlying geometric structure inherent to the data or to the function to approximate, it is often desirable to take this into account in the design of the neural network. In this work, we start with a non-autonomous ODE and build neural networks using a suitable, structure-preserving, numerical time-discretisation. The structure of the neural network is then inferred from the properties of the ODE vector field. Besides injecting more structure into the network architectures, this modelling procedure allows a better theoretical understanding of their behaviour. We present two universal approximation results and demonstrate how to impose some particular properties on the neural networks. A particular focus is on 1-Lipschitz architectures including layers that are not 1-Lipschitz. These networks are expressive and robust against adversarial attacks, as shown for the CIFAR-10 and CIFAR-100 datasets.

Related papers

• Simultaneous Weight and Architecture Optimization for Neural Networks [6.2241272327831485]
  We introduce a novel neural network training framework that transforms the process by learning architecture and parameters simultaneously with gradient descent. Central to our approach is a multi-scale encoder-decoder, in which the encoder embeds pairs of neural networks with similar functionalities close to each other. Experiments demonstrate that our framework can discover sparse and compact neural networks maintaining a high performance.
  arXiv  Detail & Related papers  (2024-10-10T19:57:36Z)
• Lipschitz constant estimation for general neural network architectures using control tools [0.05120567378386613]
  This paper is devoted to the estimation of the Lipschitz constant of neural networks using semidefinite programming. For this purpose, we interpret neural networks as time-varying dynamical systems, where the $k$-th layer corresponds to the dynamics at time $k$.
  arXiv  Detail & Related papers  (2024-05-02T09:38:16Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Set-based Neural Network Encoding Without Weight Tying [91.37161634310819]
  We propose a neural network weight encoding method for network property prediction. Our approach is capable of encoding neural networks in a model zoo of mixed architecture. We introduce two new tasks for neural network property prediction: cross-dataset and cross-architecture.
  arXiv  Detail & Related papers  (2023-05-26T04:34:28Z)
• Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis [94.64007376939735]
  We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training. We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
  arXiv  Detail & Related papers  (2022-05-11T17:43:54Z)
• Learning from Randomly Initialized Neural Network Features [24.75062551820944]
  We present the surprising result that randomly neural networks are good feature extractors in expectation. These random features correspond to finite-sample realizations of what we call Neural Network Prior Kernel (NNPK), which is inherently infinite-dimensional.
  arXiv  Detail & Related papers  (2022-02-13T23:35:11Z)
• Neural networks with linear threshold activations: structure and algorithms [1.795561427808824]
  We show that 2 hidden layers are necessary and sufficient to represent any function representable in the class. We also give precise bounds on the sizes of the neural networks required to represent any function in the class. We propose a new class of neural networks that we call shortcut linear threshold networks.
  arXiv  Detail & Related papers  (2021-11-15T22:33:52Z)
• Binary Graph Neural Networks [69.51765073772226]
  Graph Neural Networks (GNNs) have emerged as a powerful and flexible framework for representation learning on irregular data. In this paper, we present and evaluate different strategies for the binarization of graph neural networks. We show that through careful design of the models, and control of the training process, binary graph neural networks can be trained at only a moderate cost in accuracy on challenging benchmarks.
  arXiv  Detail & Related papers  (2020-12-31T18:48:58Z)
• Graph Structure of Neural Networks [104.33754950606298]
  We show how the graph structure of neural networks affect their predictive performance. A "sweet spot" of relational graphs leads to neural networks with significantly improved predictive performance. Top-performing neural networks have graph structure surprisingly similar to those of real biological neural networks.
  arXiv  Detail & Related papers  (2020-07-13T17:59:31Z)
• Deep Randomized Neural Networks [12.333836441649343]
  Randomized Neural Networks explore the behavior of neural systems where the majority of connections are fixed. This chapter surveys all the major aspects regarding the design and analysis of Randomized Neural Networks.
  arXiv  Detail & Related papers  (2020-02-27T17:57:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.