Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators

• URL: http://arxiv.org/abs/2006.11469v2
• Date: Wed, 4 Nov 2020 01:24:34 GMT
• Title: Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators
• Authors: Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama
• Abstract summary: Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. Are CF-INNs universal approximators for invertible functions? We prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes.
• Score: 72.62940905965267
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. However, their desirable characteristics such as analytic invertibility come at the cost of restricting the functional forms. This poses a question on their representation power: are CF-INNs universal approximators for invertible functions? Without a universality, there could be a well-behaved invertible transformation that the CF-INN can never approximate, hence it would render the model class unreliable. We answer this question by showing a convenient criterion: a CF-INN is universal if its layers contain affine coupling and invertible linear functions as special cases. As its corollary, we can affirmatively resolve a previously unsolved problem: whether normalizing flow models based on affine coupling can be universal distributional approximators. In the course of proving the universality, we prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes, a theoretical insight that is of interest by itself.

Related papers

• Homogeneous Artificial Neural Network [0.6526824510982799]
  The paper proposes an artificial neural network (ANN) being a global approximator for a special class of functions. The homogeneity means a symmetry of a function with respect to a group of transformations having topological characterization of a dilation.
  arXiv  Detail & Related papers  (2023-11-29T16:16:32Z)
• Equivariance with Learned Canonicalization Functions [77.32483958400282]
  We show that learning a small neural network to perform canonicalization is better than using predefineds. Our experiments show that learning the canonicalization function is competitive with existing techniques for learning equivariant functions across many tasks.
  arXiv  Detail & Related papers  (2022-11-11T21:58:15Z)
• A tradeoff between universality of equivariant models and learnability of symmetries [0.0]
  We prove that it is impossible to simultaneously learn symmetries and functions equivariant under certain conditions. We analyze certain families of neural networks for whether they satisfy the conditions of the impossibility result. On the practical side, our analysis of group-convolutional neural networks allows us generalize the well-known convolution is all you need'' to non-homogeneous spaces.
  arXiv  Detail & Related papers  (2022-10-17T21:23:22Z)
• Universal approximation property of invertible neural networks [76.95927093274392]
  Invertible neural networks (INNs) are neural network architectures with invertibility by design. Thanks to their invertibility and the tractability of Jacobian, INNs have various machine learning applications such as probabilistic modeling, generative modeling, and representation learning.
  arXiv  Detail & Related papers  (2022-04-15T10:45:26Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• Learning normal form autoencoders for data-driven discovery of universal,parameter-dependent governing equations [3.769860395223177]
  Complex systems manifest a small number of instabilities and bifurcations that are canonical in nature. Such parametric instabilities are mathematically characterized by their universal un-foldings, or normal form dynamics. We introduce deep learning autoencoders to discover coordinate transformations that capture the underlying parametric dependence.
  arXiv  Detail & Related papers  (2021-06-09T14:25:18Z)
• Approximation Theory of Tree Tensor Networks: Tensorized Multivariate Functions [0.0]
  We show that TNs can (near to) optimally replicate $h$-uniform and $h$-adaptive approximation, for any smoothness order of the target function. TNs have the capacity to (near to) optimally approximate many function classes -- without being adapted to the particular class in question.
  arXiv  Detail & Related papers  (2021-01-28T11:09:40Z)
• Universal Approximation Property of Neural Ordinary Differential Equations [19.861764482790544]
  We show that NODEs can form an $Lp$-universal approximator for continuous maps under certain conditions. We also show their stronger approximation property, namely the $sup$-universality for approximating a large class of diffeomorphisms.
  arXiv  Detail & Related papers  (2020-12-04T05:53:21Z)
• A Functional Perspective on Learning Symmetric Functions with Neural Networks [48.80300074254758]
  We study the learning and representation of neural networks defined on measures. We establish approximation and generalization bounds under different choices of regularization. The resulting models can be learned efficiently and enjoy generalization guarantees that extend across input sizes.
  arXiv  Detail & Related papers  (2020-08-16T16:34:33Z)
• UNIPoint: Universally Approximating Point Processes Intensities [125.08205865536577]
  We provide a proof that a class of learnable functions can universally approximate any valid intensity function. We implement UNIPoint, a novel neural point process model, using recurrent neural networks to parameterise sums of basis function upon each event.
  arXiv  Detail & Related papers  (2020-07-28T09:31:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.