Approximation Theory of Tree Tensor Networks: Tensorized Multivariate Functions

• URL: http://arxiv.org/abs/2101.11932v5
• Date: Tue, 25 Jun 2024 06:24:52 GMT
• Title: Approximation Theory of Tree Tensor Networks: Tensorized Multivariate Functions
• Authors: Mazen Ali, Anthony Nouy,
• Abstract summary: We show that TNs can (near to) optimally replicate $h$-uniform and $h$-adaptive approximation, for any smoothness order of the target function. TNs have the capacity to (near to) optimally approximate many function classes -- without being adapted to the particular class in question.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the approximation of multivariate functions with tensor networks (TNs). The main conclusion of this work is an answer to the following two questions: ``What are the approximation capabilities of TNs?" and "What is an appropriate model class of functions that can be approximated with TNs?" To answer the former, we show that TNs can (near to) optimally replicate $h$-uniform and $h$-adaptive approximation, for any smoothness order of the target function. Tensor networks thus exhibit universal expressivity w.r.t. isotropic, anisotropic and mixed smoothness spaces that is comparable with more general neural networks families such as deep rectified linear unit (ReLU) networks. Put differently, TNs have the capacity to (near to) optimally approximate many function classes -- without being adapted to the particular class in question. To answer the latter, as a candidate model class we consider approximation classes of TNs and show that these are (quasi-)Banach spaces, that many types of classical smoothness spaces are continuously embedded into said approximation classes and that TN approximation classes are themselves not embedded in any classical smoothness space.

Related papers

• Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classes [7.433327915285969]
  We prove the universal consistency of wide and deep ReLU neural network classifiers trained on the logistic loss. We also give sufficient conditions for a class of probability measures for which classifiers based on neural networks achieve minimax optimal rates of convergence.
  arXiv  Detail & Related papers  (2024-01-08T23:54:46Z)
• Permutation Equivariant Neural Functionals [92.0667671999604]
  This work studies the design of neural networks that can process the weights or gradients of other neural networks. We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-02-27T18:52:38Z)
• Maximally Compact and Separated Features with Regular Polytope Networks [22.376196701232388]
  We show how to extract from CNNs features the properties of emphmaximum inter-class separability and emphmaximum intra-class compactness. We obtain features similar to what can be obtained with the well-known citewen2016discriminative and other similar approaches.
  arXiv  Detail & Related papers  (2023-01-15T15:20:57Z)
• Sobolev-type embeddings for neural network approximation spaces [5.863264019032882]
  We consider neural network approximation spaces that classify functions according to the rate at which they can be approximated. We prove embedding theorems between these spaces for different values of $p$. We find that, analogous to the case of classical function spaces, it is possible to trade "smoothness" (i.e., approximation rate) for increased integrability.
  arXiv  Detail & Related papers  (2021-10-28T17:11:38Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Approximation with Tensor Networks. Part II: Approximation Rates for Smoothness Classes [0.0]
  We study the approximation by tensor networks (TNs) of functions from smoothness classes. The resulting tool can be interpreted as a feed-forward neural network. We show that arbitrary Besov functions can be approximated with optimal or near to optimal rate.
  arXiv  Detail & Related papers  (2020-06-30T21:57:42Z)
• Approximation with Tensor Networks. Part I: Approximation Spaces [0.0]
  We study the approximation of functions by tensor networks (TNs) We show that Lebesgue $Lp$-spaces in one dimension can be identified with tensor product spaces of arbitrary order through tensorization. We show that functions in these approximation classes do not possess any Besov smoothness.
  arXiv  Detail & Related papers  (2020-06-30T21:32:59Z)
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators [72.62940905965267]
  Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. Are CF-INNs universal approximators for invertible functions? We prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes.
  arXiv  Detail & Related papers  (2020-06-20T02:07:37Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)
• Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach [85.12934750565971]
  Efficient modelling of feature interactions underpins supervised learning for non-sequential tasks. To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor. For enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors.
  arXiv  Detail & Related papers  (2020-01-27T22:38:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.