Function Approximation via Sparse Random Features
- URL: http://arxiv.org/abs/2103.03191v1
- Date: Thu, 4 Mar 2021 17:53:54 GMT
- Title: Function Approximation via Sparse Random Features
- Authors: Abolfazl Hashemi, Hayden Schaeffer, Robert Shi, Ufuk Topcu, Giang
Tran, Rachel Ward
- Abstract summary: This paper introduces the sparse random feature method that learns parsimonious random feature models utilizing techniques from compressive sensing.
We show that the sparse random feature method outperforms shallow networks for well-structured functions and applications to scientific machine learning tasks.
- Score: 23.325877475827337
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Random feature methods have been successful in various machine learning
tasks, are easy to compute, and come with theoretical accuracy bounds. They
serve as an alternative approach to standard neural networks since they can
represent similar function spaces without a costly training phase. However, for
accuracy, random feature methods require more measurements than trainable
parameters, limiting their use for data-scarce applications or problems in
scientific machine learning. This paper introduces the sparse random feature
method that learns parsimonious random feature models utilizing techniques from
compressive sensing. We provide uniform bounds on the approximation error for
functions in a reproducing kernel Hilbert space depending on the number of
samples and the distribution of features. The error bounds improve with
additional structural conditions, such as coordinate sparsity, compact clusters
of the spectrum, or rapid spectral decay. We show that the sparse random
feature method outperforms shallow networks for well-structured functions and
applications to scientific machine learning tasks.
Related papers
- Variance-Reducing Couplings for Random Features: Perspectives from Optimal Transport [57.73648780299374]
Random features (RFs) are a popular technique to scale up kernel methods in machine learning, replacing exact kernel evaluations with Monte Carlo estimates.
We tackle this through the unifying framework of optimal transport, using theoretical insights and numerical algorithms to develop novel, high-performing RF couplings for kernels defined on Euclidean and discrete input spaces.
We reach surprising conclusions about the benefits and limitations of variance reduction as a paradigm.
arXiv Detail & Related papers (2024-05-26T12:25:09Z) - D2NO: Efficient Handling of Heterogeneous Input Function Spaces with
Distributed Deep Neural Operators [7.119066725173193]
We propose a novel distributed approach to deal with input functions that exhibit heterogeneous properties.
A central neural network is used to handle shared information across all output functions.
We demonstrate that the corresponding neural network is a universal approximator of continuous nonlinear operators.
arXiv Detail & Related papers (2023-10-29T03:29:59Z) - Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization [73.80101701431103]
The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks.
We study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility.
arXiv Detail & Related papers (2023-04-17T14:23:43Z) - Score-based Diffusion Models in Function Space [140.792362459734]
Diffusion models have recently emerged as a powerful framework for generative modeling.
We introduce a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space.
We show that the corresponding discretized algorithm generates accurate samples at a fixed cost independent of the data resolution.
arXiv Detail & Related papers (2023-02-14T23:50:53Z) - Hybrid Random Features [60.116392415715275]
We propose a new class of random feature methods for linearizing softmax and Gaussian kernels called hybrid random features (HRFs)
HRFs automatically adapt the quality of kernel estimation to provide most accurate approximation in the defined regions of interest.
arXiv Detail & Related papers (2021-10-08T20:22:59Z) - Large-Scale Learning with Fourier Features and Tensor Decompositions [3.6930948691311007]
We exploit the tensor product structure of deterministic Fourier features, which enables us to represent the model parameters as a low-rank tensor decomposition.
We demonstrate by means of numerical experiments how our low-rank tensor approach obtains the same performance of the corresponding nonparametric model.
arXiv Detail & Related papers (2021-09-03T14:12:53Z) - Random features for adaptive nonlinear control and prediction [15.354147587211031]
We propose a tractable algorithm for both adaptive control and adaptive prediction.
We approximate the unknown dynamics with a finite expansion in $textitrandom$ basis functions.
Remarkably, our explicit bounds only depend $textitpolynomially$ on the underlying parameters of the system.
arXiv Detail & Related papers (2021-06-07T13:15:40Z) - A Simple and General Debiased Machine Learning Theorem with Finite
Sample Guarantees [4.55274575362193]
We provide a nonasymptotic debiased machine learning theorem that encompasses any global or local functional of any machine learning algorithm.
Our results culminate in a simple set of conditions that an analyst can use to translate modern learning theory rates into traditional statistical inference.
arXiv Detail & Related papers (2021-05-31T17:57:02Z) - Sparse Spectrum Warped Input Measures for Nonstationary Kernel Learning [29.221457769884648]
We propose a general form of explicit, input-dependent, measure-valued warpings for learning nonstationary kernels.
The proposed learning algorithm warps inputs as conditional Gaussian measures that control the smoothness of a standard stationary kernel.
We demonstrate a remarkable efficiency in the number of parameters of the warping functions in learning problems with both small and large data regimes.
arXiv Detail & Related papers (2020-10-09T01:10:08Z) - A Functional Perspective on Learning Symmetric Functions with Neural
Networks [48.80300074254758]
We study the learning and representation of neural networks defined on measures.
We establish approximation and generalization bounds under different choices of regularization.
The resulting models can be learned efficiently and enjoy generalization guarantees that extend across input sizes.
arXiv Detail & Related papers (2020-08-16T16:34:33Z) - UNIPoint: Universally Approximating Point Processes Intensities [125.08205865536577]
We provide a proof that a class of learnable functions can universally approximate any valid intensity function.
We implement UNIPoint, a novel neural point process model, using recurrent neural networks to parameterise sums of basis function upon each event.
arXiv Detail & Related papers (2020-07-28T09:31:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.