Forward and Inverse Approximation Theory for Linear Temporal Convolutional Networks

• URL: http://arxiv.org/abs/2305.18478v1
• Date: Mon, 29 May 2023 11:08:04 GMT
• Title: Forward and Inverse Approximation Theory for Linear Temporal Convolutional Networks
• Authors: Haotian Jiang, Qianxiao Li
• Abstract summary: We prove an approximation rate estimate (Jackson-type result) and an inverse approximation theorem (Bernstein-type result) We provide a comprehensive characterization of the types of sequential relationships that can be efficiently captured by a temporal convolutional architecture.
• Score: 20.9427668489352
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a theoretical analysis of the approximation properties of convolutional architectures when applied to the modeling of temporal sequences. Specifically, we prove an approximation rate estimate (Jackson-type result) and an inverse approximation theorem (Bernstein-type result), which together provide a comprehensive characterization of the types of sequential relationships that can be efficiently captured by a temporal convolutional architecture. The rate estimate improves upon a previous result via the introduction of a refined complexity measure, whereas the inverse approximation theorem is new.

Related papers

• Approximation Rate of the Transformer Architecture for Sequence Modeling [18.166959969957315]
  We consider a class of non-linear relationships and identify a novel notion of complexity measures to establish an explicit Jackson-type approximation rate estimate for the Transformer. This rate reveals the structural properties of the Transformer and suggests the types of sequential relationships it is best suited for approximating.
  arXiv  Detail & Related papers  (2023-05-29T10:56:36Z)
• A predictive physics-aware hybrid reduced order model for reacting flows [65.73506571113623]
  A new hybrid predictive Reduced Order Model (ROM) is proposed to solve reacting flow problems. The number of degrees of freedom is reduced from thousands of temporal points to a few POD modes with their corresponding temporal coefficients. Two different deep learning architectures have been tested to predict the temporal coefficients.
  arXiv  Detail & Related papers  (2023-01-24T08:39:20Z)
• Connecting Optimization and Generalization via Gradient Flow Path Length [17.099964074542818]
  We propose a framework to connect optimization with generalization by analyzing the generalization error based on the length of optimization trajectory under the gradient flow algorithm after convergence. Our framework can be applied to broad settings. For example, we use it to obtain generalization estimates on three distinct machine learning models.
  arXiv  Detail & Related papers  (2022-02-22T04:58:16Z)
• Approximation Theory of Convolutional Architectures for Time Series Modelling [15.42770933459534]
  We study the approximation properties of convolutional architectures applied to time series modelling. Recent results reveal an intricate connection between approximation efficiency and memory structures in the data generation process.
  arXiv  Detail & Related papers  (2021-07-20T09:19:26Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)
• Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases [131.50372468579067]
  We represent the reconstructed surface as an atlas, using a neural network. We empirically show that our method achieves results that exceed that state of the art in the accuracy of unsupervised correspondences and accuracy of surface reconstruction.
  arXiv  Detail & Related papers  (2021-04-14T16:21:22Z)
• Joint Network Topology Inference via Structured Fusion Regularization [70.30364652829164]
  Joint network topology inference represents a canonical problem of learning multiple graph Laplacian matrices from heterogeneous graph signals. We propose a general graph estimator based on a novel structured fusion regularization. We show that the proposed graph estimator enjoys both high computational efficiency and rigorous theoretical guarantee.
  arXiv  Detail & Related papers  (2021-03-05T04:42:32Z)
• Posterior-Aided Regularization for Likelihood-Free Inference [23.708122045184698]
  Posterior-Aided Regularization (PAR) is applicable to learning the density estimator, regardless of the model structure. We provide a unified estimation method of PAR to estimate both reverse KL term and mutual information term with a single neural network.
  arXiv  Detail & Related papers  (2021-02-15T16:59:30Z)
• Supporting Optimal Phase Space Reconstructions Using Neural Network Architecture for Time Series Modeling [68.8204255655161]
  We propose an artificial neural network with a mechanism to implicitly learn the phase spaces properties. Our approach is either as competitive as or better than most state-of-the-art strategies.
  arXiv  Detail & Related papers  (2020-06-19T21:04:47Z)
• Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives [97.16266088683061]
  The article rigorously establishes why symplectic discretization schemes are important for momentum-based optimization algorithms. It provides a characterization of algorithms that exhibit accelerated convergence.
  arXiv  Detail & Related papers  (2020-02-28T00:32:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.