Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nystr\"om Method, and Use of Kernels in Machine Learning: Tutorial and Survey

• URL: http://arxiv.org/abs/2106.08443v1
• Date: Tue, 15 Jun 2021 21:29:12 GMT
• Title: Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nystr\"om Method, and Use of Kernels in Machine Learning: Tutorial and Survey
• Authors: Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, Mark Crowley
• Abstract summary: We start with reviewing the history of kernels in functional analysis and machine learning. We introduce types of use of kernels in machine learning including kernel methods, kernel learning by semi-definite programming, Hilbert-Schmidt independence criterion, maximum mean discrepancy, kernel mean embedding, and kernel dimensionality reduction. This paper can be useful for various fields of science including machine learning, dimensionality reduction, functional analysis in mathematics, and mathematical physics in quantum mechanics.
• Score: 5.967999555890417
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This is a tutorial and survey paper on kernels, kernel methods, and related fields. We start with reviewing the history of kernels in functional analysis and machine learning. Then, Mercer kernel, Hilbert and Banach spaces, Reproducing Kernel Hilbert Space (RKHS), Mercer's theorem and its proof, frequently used kernels, kernel construction from distance metric, important classes of kernels (including bounded, integrally positive definite, universal, stationary, and characteristic kernels), kernel centering and normalization, and eigenfunctions are explained in detail. Then, we introduce types of use of kernels in machine learning including kernel methods (such as kernel support vector machines), kernel learning by semi-definite programming, Hilbert-Schmidt independence criterion, maximum mean discrepancy, kernel mean embedding, and kernel dimensionality reduction. We also cover rank and factorization of kernel matrix as well as the approximation of eigenfunctions and kernels using the Nystr{\"o}m method. This paper can be useful for various fields of science including machine learning, dimensionality reduction, functional analysis in mathematics, and mathematical physics in quantum mechanics.

Related papers

• Spectral Truncation Kernels: Noncommutativity in $C^*$-algebraic Kernel Machines [12.11705128358537]
  We propose a new class of positive definite kernels based on the spectral truncation. We show that it is a governing factor leading to performance enhancement. We also propose a deep learning perspective to increase the representation capacity of spectral truncation kernels.
  arXiv  Detail & Related papers  (2024-05-28T04:47:12Z)
• Geometric Learning with Positively Decomposable Kernels [6.5497574505866885]
  We propose the use of reproducing kernel Krein space (RKKS) based methods, which require only kernels that admit a positive decomposition. We show that one does not need to access this decomposition in order to learn in RKKS.
  arXiv  Detail & Related papers  (2023-10-20T21:18:04Z)
• Reproducing kernel Hilbert spaces in the mean field limit [6.844996517347866]
  kernels are function spaces generated by kernels, so called reproducing kernel Hilbert spaces. We show the rigorous mean field limit of kernels and provide a detailed analysis of the limiting reproducing Hilbert space.
  arXiv  Detail & Related papers  (2023-02-28T09:46:44Z)
• Geometry-aware Bayesian Optimization in Robotics using Riemannian Mat\'ern Kernels [64.62221198500467]
  We show how to implement geometry-aware kernels for Bayesian optimization. This technique can be used for control parameter tuning, parametric policy adaptation, and structure design in robotics.
  arXiv  Detail & Related papers  (2021-11-02T09:47:22Z)
• Fast Sketching of Polynomial Kernels of Polynomial Degree [61.83993156683605]
  kernel is especially important as other kernels can often be approximated by the kernel via a Taylor series expansion. Recent techniques in sketching reduce the dependence in the running time on the degree oblivious $q$ of the kernel. We give a new sketch which greatly improves upon this running time, by removing the dependence on $q$ in the leading order term.
  arXiv  Detail & Related papers  (2021-08-21T02:14:55Z)
• Kernel Continual Learning [117.79080100313722]
  kernel continual learning is a simple but effective variant of continual learning to tackle catastrophic forgetting. episodic memory unit stores a subset of samples for each task to learn task-specific classifiers based on kernel ridge regression. variational random features to learn a data-driven kernel for each task.
  arXiv  Detail & Related papers  (2021-07-12T22:09:30Z)
• Taming Nonconvexity in Kernel Feature Selection---Favorable Properties of the Laplace Kernel [77.73399781313893]
  A challenge is to establish the objective function of kernel-based feature selection. The gradient-based algorithms available for non-global optimization are only able to guarantee convergence to local minima.
  arXiv  Detail & Related papers  (2021-06-17T11:05:48Z)
• Kernel Identification Through Transformers [54.3795894579111]
  Kernel selection plays a central role in determining the performance of Gaussian Process (GP) models. This work addresses the challenge of constructing custom kernel functions for high-dimensional GP regression models. We introduce a novel approach named KITT: Kernel Identification Through Transformers.
  arXiv  Detail & Related papers  (2021-06-15T14:32:38Z)
• Covariant quantum kernels for data with group structure [1.51714450051254]
  We introduce a class of quantum kernels that can be used for data with a group structure. We apply this method to a learning problem on a coset-space that embodies the structure of many essential learning problems on groups.
  arXiv  Detail & Related papers  (2021-05-07T17:38:58Z)
• Faster Kernel Matrix Algebra via Density Estimation [46.253698241653254]
  We study fast algorithms for computing fundamental properties of a positive semidefinite kernel matrix $K in mathbbRn times n$ corresponding to $n$ points.
  arXiv  Detail & Related papers  (2021-02-16T18:25:47Z)
• Lecture notes: Efficient approximation of kernel functions [4.177892889752434]
  Notes endeavour to collect in one place the mathematical background required to understand the properties of kernels in general. We briefly motivate the use of kernels in Machine Learning with the example of the support vector machine. After a brief discussion of Hilbert spaces, including the Reproducing Kernel Hilbert Space construction, we present Mercer's theorem.
  arXiv  Detail & Related papers  (2020-05-04T15:30:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.