Fast convolution algorithm for state space models

• URL: http://arxiv.org/abs/2411.17729v2
• Date: Fri, 29 Nov 2024 01:10:55 GMT
• Title: Fast convolution algorithm for state space models
• Authors: Gregory Beylkin,
• Abstract summary: We present a robust algorithm for applying a matrix transfer function of a linear time invariant system (LTI) in time domain. We demonstrate that, for any finite user-selected accuracy, the number of matrix-vector multiplications can be reduced to $mathcalOleft(log_2Lright)$. Importantly, using a cascade implementation in time domain, applying the transfer function requires only $N+1$ matrix-vector multiplications.
• Score: 0.0
• License:
• Abstract: We present a fast, robust algorithm for applying a matrix transfer function of a linear time invariant system (LTI) in time domain. Computing $L$ states of a multiple-input multiple-output (MIMO) LTI appears to require $L$ matrix-vector multiplications. We demonstrate that, for any finite user-selected accuracy, the number of matrix-vector multiplications can be reduced to $\mathcal{O}\left(\log_{2}L\right)$ (within an $\mathcal{O}\left(L\right)$ algorithm). The algorithm uses an approximation of the rational transfer function in the z-domain by a matrix polynomial of degree $2^{N+1}-1$, where $N$ is chosen to achieve any user-selected accuracy. Importantly, using a cascade implementation in time domain, applying the transfer function requires only $N+1$ matrix-vector multiplications. We note that LTI systems are used in state space models (SSMs) for modeling long range dependencies where $L$ is large. In applications where the state matrix of LTI system is approximated by a structured matrix, the computational cost is further reduced. We briefly describe several structured approximations of matrices that can be used for such purpose.

Related papers

• Faster Sampling from Log-Concave Densities over Polytopes via Efficient Linear Solvers [29.212403229351253]
  We present a nearly-optimal implementation of this Markov chain with per-step complexity which is roughly the number of non-zero entries of $A$ while the number of Markov chain steps remains the same. Key technical ingredients are 1) to show that the matrices that arise in this Dikin walk change slowly, 2) to deploy efficient linear solvers that can leverage this slow change to speed up matrix inversion by using information computed in previous steps, and 3) to speed up the computation of the determinantal term in the Metropolis filter step via a randomized Taylor series-based estimator.
  arXiv  Detail & Related papers  (2024-09-06T14:49:43Z)
• Fast Matrix Multiplication Without Tears: A Constraint Programming Approach [8.52818380743467]
  It is known that the multiplication of an $N times M$ matrix with an $M times P$ matrix can be performed using fewer multiplications than what the naive $NMP approach suggests. This gives rise to the constraint satisfaction problem of fast matrix multiplication. We propose a simple yet novel Constraint Programming approach to find non-commutative algorithms for fast matrix multiplication.
  arXiv  Detail & Related papers  (2023-06-01T19:15:24Z)
• Refined Regret for Adversarial MDPs with Linear Function Approximation [50.00022394876222]
  We consider learning in an adversarial Decision Process (MDP) where the loss functions can change arbitrarily over $K$ episodes. This paper provides two algorithms that improve the regret to $tildemathcal O(K2/3)$ in the same setting.
  arXiv  Detail & Related papers  (2023-01-30T14:37:21Z)
• Fast Differentiable Matrix Square Root [65.67315418971688]
  We propose two more efficient variants to compute the differentiable matrix square root. For the forward propagation, one method is to use Matrix Taylor Polynomial (MTP) The other method is to use Matrix Pad'e Approximants (MPA)
  arXiv  Detail & Related papers  (2022-01-21T12:18:06Z)
• Efficient Matrix-Free Approximations of Second-Order Information, with Applications to Pruning and Optimization [16.96639526117016]
  We investigate matrix-free, linear-time approaches for estimating Inverse-Hessian Vector Products (IHVPs) These algorithms yield state-of-the-art results for network pruning and optimization with lower computational overhead relative to existing second-order methods.
  arXiv  Detail & Related papers  (2021-07-07T17:01:34Z)
• Non-PSD Matrix Sketching with Applications to Regression and Optimization [56.730993511802865]
  We present dimensionality reduction methods for non-PSD and square-roots" matrices. We show how these techniques can be used for multiple downstream tasks.
  arXiv  Detail & Related papers  (2021-06-16T04:07:48Z)
• Quantum algorithms for spectral sums [50.045011844765185]
  We propose new quantum algorithms for estimating spectral sums of positive semi-definite (PSD) matrices. We show how the algorithms and techniques used in this work can be applied to three problems in spectral graph theory.
  arXiv  Detail & Related papers  (2020-11-12T16:29:45Z)
• Linear-Sample Learning of Low-Rank Distributions [56.59844655107251]
  We show that learning $ktimes k$, rank-$r$, matrices to normalized $L_1$ distance requires $Omega(frackrepsilon2)$ samples. We propose an algorithm that uses $cal O(frackrepsilon2log2fracepsilon)$ samples, a number linear in the high dimension, and nearly linear in the matrices, typically low, rank proofs.
  arXiv  Detail & Related papers  (2020-09-30T19:10:32Z)
• What if Neural Networks had SVDs? [66.91160214071088]
  Various Neural Networks employ time-consuming matrix operations like matrix inversion. We present an algorithm that is fast enough to speed up several matrix operations.
  arXiv  Detail & Related papers  (2020-09-29T12:58:52Z)
• Approximate Multiplication of Sparse Matrices with Limited Space [24.517908972536432]
  We develop sparse co-occuring directions, which reduces the time complexity to $widetildeOleft((nnz(X)+nnz(Y))ell+nell2right)$ in expectation. Theoretical analysis reveals that the approximation error of our algorithm is almost the same as that of COD.
  arXiv  Detail & Related papers  (2020-09-08T05:39:19Z)
• Fast Matrix Square Roots with Applications to Gaussian Processes and Bayesian Optimization [24.085358075449406]
  Matrix square roots and their inverses arise frequently in machine learning. We introduce a highly-efficient quadratic-time algorithm for computing $mathbf K1/2 mathbf b$, $mathbf K-1/2 mathbf b$, and their derivatives through matrix-vector multiplication (MVMs) Our method combines Krylov subspace methods with a rational approximation and typically $4$ decimal places of accuracy with fewer than $100$ MVMs.
  arXiv  Detail & Related papers  (2020-06-19T17:56:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.