Function Approximation via Sparse Random Features

• URL: http://arxiv.org/abs/2103.03191v1
• Date: Thu, 4 Mar 2021 17:53:54 GMT
• Title: Function Approximation via Sparse Random Features
• Authors: Abolfazl Hashemi, Hayden Schaeffer, Robert Shi, Ufuk Topcu, Giang Tran, Rachel Ward
• Abstract summary: This paper introduces the sparse random feature method that learns parsimonious random feature models utilizing techniques from compressive sensing. We show that the sparse random feature method outperforms shallow networks for well-structured functions and applications to scientific machine learning tasks.
• Score: 23.325877475827337
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Random feature methods have been successful in various machine learning tasks, are easy to compute, and come with theoretical accuracy bounds. They serve as an alternative approach to standard neural networks since they can represent similar function spaces without a costly training phase. However, for accuracy, random feature methods require more measurements than trainable parameters, limiting their use for data-scarce applications or problems in scientific machine learning. This paper introduces the sparse random feature method that learns parsimonious random feature models utilizing techniques from compressive sensing. We provide uniform bounds on the approximation error for functions in a reproducing kernel Hilbert space depending on the number of samples and the distribution of features. The error bounds improve with additional structural conditions, such as coordinate sparsity, compact clusters of the spectrum, or rapid spectral decay. We show that the sparse random feature method outperforms shallow networks for well-structured functions and applications to scientific machine learning tasks.

Related papers

• Operator Learning Using Random Features: A Tool for Scientific Computing [3.745868534225104]
  Supervised operator learning centers on the use of training data to estimate maps between infinite-dimensional spaces. This paper introduces the function-valued random features method. It leads to a supervised operator learning architecture that is practical for nonlinear problems.
  arXiv  Detail & Related papers  (2024-08-12T23:10:39Z)
• Universal approximation property of Banach space-valued random feature models including random neural networks [3.3379026542599934]
  We introduce a Banach space-valued extension of random feature learning. By randomly initializing the feature maps, only the linear readout needs to be trained. We derive approximation rates and an explicit algorithm to learn an element of the given Banach space.
  arXiv  Detail & Related papers  (2023-12-13T11:27:15Z)
• D2NO: Efficient Handling of Heterogeneous Input Function Spaces with Distributed Deep Neural Operators [7.119066725173193]
  We propose a novel distributed approach to deal with input functions that exhibit heterogeneous properties. A central neural network is used to handle shared information across all output functions. We demonstrate that the corresponding neural network is a universal approximator of continuous nonlinear operators.
  arXiv  Detail & Related papers  (2023-10-29T03:29:59Z)
• Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization [73.80101701431103]
  The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. We study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility.
  arXiv  Detail & Related papers  (2023-04-17T14:23:43Z)
• Score-based Diffusion Models in Function Space [140.792362459734]
  Diffusion models have recently emerged as a powerful framework for generative modeling. We introduce a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space. We show that the corresponding discretized algorithm generates accurate samples at a fixed cost independent of the data resolution.
  arXiv  Detail & Related papers  (2023-02-14T23:50:53Z)
• Monte Carlo Neural PDE Solver for Learning PDEs via Probabilistic Representation [59.45669299295436]
  We propose a Monte Carlo PDE solver for training unsupervised neural solvers. We use the PDEs' probabilistic representation, which regards macroscopic phenomena as ensembles of random particles. Our experiments on convection-diffusion, Allen-Cahn, and Navier-Stokes equations demonstrate significant improvements in accuracy and efficiency.
  arXiv  Detail & Related papers  (2023-02-10T08:05:19Z)
• Hybrid Random Features [60.116392415715275]
  We propose a new class of random feature methods for linearizing softmax and Gaussian kernels called hybrid random features (HRFs) HRFs automatically adapt the quality of kernel estimation to provide most accurate approximation in the defined regions of interest.
  arXiv  Detail & Related papers  (2021-10-08T20:22:59Z)
• Large-Scale Learning with Fourier Features and Tensor Decompositions [3.6930948691311007]
  We exploit the tensor product structure of deterministic Fourier features, which enables us to represent the model parameters as a low-rank tensor decomposition. We demonstrate by means of numerical experiments how our low-rank tensor approach obtains the same performance of the corresponding nonparametric model.
  arXiv  Detail & Related papers  (2021-09-03T14:12:53Z)
• Sparse Spectrum Warped Input Measures for Nonstationary Kernel Learning [29.221457769884648]
  We propose a general form of explicit, input-dependent, measure-valued warpings for learning nonstationary kernels. The proposed learning algorithm warps inputs as conditional Gaussian measures that control the smoothness of a standard stationary kernel. We demonstrate a remarkable efficiency in the number of parameters of the warping functions in learning problems with both small and large data regimes.
  arXiv  Detail & Related papers  (2020-10-09T01:10:08Z)
• A Functional Perspective on Learning Symmetric Functions with Neural Networks [48.80300074254758]
  We study the learning and representation of neural networks defined on measures. We establish approximation and generalization bounds under different choices of regularization. The resulting models can be learned efficiently and enjoy generalization guarantees that extend across input sizes.
  arXiv  Detail & Related papers  (2020-08-16T16:34:33Z)
• UNIPoint: Universally Approximating Point Processes Intensities [125.08205865536577]
  We provide a proof that a class of learnable functions can universally approximate any valid intensity function. We implement UNIPoint, a novel neural point process model, using recurrent neural networks to parameterise sums of basis function upon each event.
  arXiv  Detail & Related papers  (2020-07-28T09:31:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.