Continuous Generative Neural Networks

• URL: http://arxiv.org/abs/2205.14627v2
• Date: Thu, 20 Apr 2023 09:41:04 GMT
• Title: Continuous Generative Neural Networks
• Authors: Giovanni S. Alberti, Matteo Santacesaria and Silvia Sciutto
• Abstract summary: We study Continuous Generative Neural Networks (CGNNs) in the continuous setting. The architecture is inspired by DCGAN, with one fully connected layer, several convolutional layers and nonlinear activation functions. We present conditions on the convolutional filters and on the nonlinearity that guarantee that a CGNN is injective.
• Score: 0.966840768820136
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we present and study Continuous Generative Neural Networks (CGNNs), namely, generative models in the continuous setting: the output of a CGNN belongs to an infinite-dimensional function space. The architecture is inspired by DCGAN, with one fully connected layer, several convolutional layers and nonlinear activation functions. In the continuous $L^2$ setting, the dimensions of the spaces of each layer are replaced by the scales of a multiresolution analysis of a compactly supported wavelet. We present conditions on the convolutional filters and on the nonlinearity that guarantee that a CGNN is injective. This theory finds applications to inverse problems, and allows for deriving Lipschitz stability estimates for (possibly nonlinear) infinite-dimensional inverse problems with unknowns belonging to the manifold generated by a CGNN. Several numerical simulations, including signal deblurring, illustrate and validate this approach.

Related papers

• Deep Learning as Ricci Flow [38.27936710747996]
  Deep neural networks (DNNs) are powerful tools for approximating the distribution of complex data. We show that the transformations performed by DNNs during classification tasks have parallels to those expected under Hamilton's Ricci flow. Our findings motivate the use of tools from differential and discrete geometry to the problem of explainability in deep learning.
  arXiv  Detail & Related papers  (2024-04-22T15:12:47Z)
• Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity [1.2891210250935148]
  We show that for every stably converged NN of continuous activation functions, its weight matrix encodes a continuous function that approximates its training dataset to within a finite margin of error over a bounded domain. Our results have implications for understanding how NNs break the curse of dimensionality by harnessing memory capacity for expressivity. This Layer Matrix Decomposition (LMD) further suggests a close relationship between eigen-decomposition of NN layers and the latest advances in conceptualizations of Hopfield networks and Transformer NN models.
  arXiv  Detail & Related papers  (2023-09-12T05:36:08Z)
• Tangent Bundle Convolutional Learning: from Manifolds to Cellular Sheaves and Back [84.61160272624262]
  We define tangent bundle filters and tangent bundle neural networks (TNNs) based on this convolution operation. Tangent bundle filters admit a spectral representation that generalizes the ones of scalar manifold filters, graph filters and standard convolutional filters in continuous time. We numerically evaluate the effectiveness of the proposed architecture on various learning tasks.
  arXiv  Detail & Related papers  (2023-03-20T17:57:15Z)
• Tangent Bundle Filters and Neural Networks: from Manifolds to Cellular Sheaves and Back [114.01902073621577]
  We use the convolution to define tangent bundle filters and tangent bundle neural networks (TNNs) We discretize TNNs both in space and time domains, showing that their discrete counterpart is a principled variant of the recently introduced Sheaf Neural Networks. We numerically evaluate the effectiveness of the proposed architecture on a denoising task of a tangent vector field over the unit 2-sphere.
  arXiv  Detail & Related papers  (2022-10-26T21:55:45Z)
• Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis [3.5450828190071655]
  Causal operators (COs) play a central role in contemporary analysis. There is still no canonical framework for designing Deep Learning (DL) models capable of approximating COs. This paper proposes a "geometry-aware" solution to this open problem by introducing a DL model-design framework.
  arXiv  Detail & Related papers  (2022-10-24T14:43:03Z)
• Exploring Linear Feature Disentanglement For Neural Networks [63.20827189693117]
  Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have achieved great success in neural networks (NNs) Due to the complex non-linear characteristic of samples, the objective of those activation functions is to project samples from their original feature space to a linear separable feature space. This phenomenon ignites our interest in exploring whether all features need to be transformed by all non-linear functions in current typical NNs.
  arXiv  Detail & Related papers  (2022-03-22T13:09:17Z)
• Stability of Neural Networks on Manifolds to Relative Perturbations [118.84154142918214]
  Graph Neural Networks (GNNs) show impressive performance in many practical scenarios. GNNs can scale well on large size graphs, but this is contradicted by the fact that existing stability bounds grow with the number of nodes.
  arXiv  Detail & Related papers  (2021-10-10T04:37:19Z)
• Convolutional Filtering and Neural Networks with Non Commutative Algebras [153.20329791008095]
  We study the generalization of non commutative convolutional neural networks. We show that non commutative convolutional architectures can be stable to deformations on the space of operators.
  arXiv  Detail & Related papers  (2021-08-23T04:22:58Z)
• Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design [2.752441514346229]
  Vanishing and exploding gradients during weight optimization through backpropagation can be difficult to train. We propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretization of continuous-time Hamiltonian systems. Our main result is that a broad set of H-DNNs ensures non-vanishing gradients by design for an arbitrary network depth. The good performance of H-DNNs is demonstrated on benchmark classification problems, including image classification with the MNIST dataset.
  arXiv  Detail & Related papers  (2021-05-27T14:52:22Z)
• Uncertainty Propagation in Convolutional Neural Networks: Technical Report [13.224071661974593]
  We study the problem of propagation of uncertainty through typical building blocks of a Convolutional Neural Network (CNN) These include layers that perform linear operations, as well as layers that act non-linearly on their input, such as the Rectified Linear Unit (ReLU)
  arXiv  Detail & Related papers  (2021-02-11T15:17:46Z)
• Nonlinear State-Space Generalizations of Graph Convolutional Neural Networks [172.18295279061607]
  Graph convolutional neural networks (GCNNs) learn compositional representations from network data by nesting linear graph convolutions into nonlinearities. In this work, we approach GCNNs from a state-space perspective revealing that the graph convolutional module is a minimalistic linear state-space model. We show that this state update may be problematic because it is nonparametric, and depending on the graph spectrum it may explode or vanish. We propose a novel family of nodal aggregation rules that aggregate node features within a layer in a nonlinear state-space parametric fashion allowing for a better trade-off.
  arXiv  Detail & Related papers  (2020-10-27T19:48:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.