Kernel Methods and their derivatives: Concept and perspectives for the Earth system sciences

• URL: http://arxiv.org/abs/2007.14706v2
• Date: Mon, 5 Oct 2020 16:18:51 GMT
• Title: Kernel Methods and their derivatives: Concept and perspectives for the Earth system sciences
• Authors: J. Emmanuel Johnson, Valero Laparra, Adri\'an P\'erez-Suay, Miguel D. Mahecha and Gustau Camps-Valls
• Abstract summary: We show that it is possible to interpret the functions learned by various kernel methods is intuitive despite their complexity. Specifically, we show that derivatives of these functions have a simple mathematical formulation, are easy to compute, and can be applied to many different problems.
• Score: 8.226445359788402
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Kernel methods are powerful machine learning techniques which implement generic non-linear functions to solve complex tasks in a simple way. They Have a solid mathematical background and exhibit excellent performance in practice. However, kernel machines are still considered black-box models as the feature mapping is not directly accessible and difficult to interpret.The aim of this work is to show that it is indeed possible to interpret the functions learned by various kernel methods is intuitive despite their complexity. Specifically, we show that derivatives of these functions have a simple mathematical formulation, are easy to compute, and can be applied to many different problems. We note that model function derivatives in kernel machines is proportional to the kernel function derivative. We provide the explicit analytic form of the first and second derivatives of the most common kernel functions with regard to the inputs as well as generic formulas to compute higher order derivatives. We use them to analyze the most used supervised and unsupervised kernel learning methods: Gaussian Processes for regression, Support Vector Machines for classification, Kernel Entropy Component Analysis for density estimation, and the Hilbert-Schmidt Independence Criterion for estimating the dependency between random variables. For all cases we expressed the derivative of the learned function as a linear combination of the kernel function derivative. Moreover we provide intuitive explanations through illustrative toy examples and show how to improve the interpretation of real applications in the context of spatiotemporal Earth system data cubes. This work reflects on the observation that function derivatives may play a crucial role in kernel methods analysis and understanding.

Related papers

• Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization [73.80101701431103]
  The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. We study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility.
  arXiv  Detail & Related papers  (2023-04-17T14:23:43Z)
• An Introduction to Kernel and Operator Learning Methods for Homogenization by Self-consistent Clustering Analysis [0.48747801442240574]
  The article presents a thorough analysis on the mathematical underpinnings of the operator learning paradigm. The proposed kernel operator learning method uses graph kernel networks to come up with a mechanistic reduced order method for multiscale homogenization.
  arXiv  Detail & Related papers  (2022-12-01T02:36:16Z)
• NeuralEF: Deconstructing Kernels by Deep Neural Networks [47.54733625351363]
  Traditional nonparametric solutions based on the Nystr"om formula suffer from scalability issues. Recent work has resorted to a parametric approach, i.e., training neural networks to approximate the eigenfunctions. We show that these problems can be fixed by using a new series of objective functions that generalizes to space of supervised and unsupervised learning problems.
  arXiv  Detail & Related papers  (2022-04-30T05:31:07Z)
• AutoIP: A United Framework to Integrate Physics into Gaussian Processes [15.108333340471034]
  We propose a framework that can integrate all kinds of differential equations into Gaussian processes. Our method shows improvement upon vanilla GPs in both simulation and several real-world applications.
  arXiv  Detail & Related papers  (2022-02-24T19:02:14Z)
• Adjoint-aided inference of Gaussian process driven differential equations [0.8257490175399691]
  We show how the adjoint of a linear system can be used to efficiently infer forcing functions modelled as GPs. We demonstrate the approach on systems of both ordinary and partial differential equations.
  arXiv  Detail & Related papers  (2022-02-09T17:35:14Z)
• Vector-valued Gaussian Processes on Riemannian Manifolds via Gauge Equivariant Projected Kernels [108.60991563944351]
  We present a recipe for constructing gauge equivariant kernels, which induce vector-valued Gaussian processes coherent with geometry. We extend standard Gaussian process training methods, such as variational inference, to this setting.
  arXiv  Detail & Related papers  (2021-10-27T13:31:10Z)
• Advanced Stationary and Non-Stationary Kernel Designs for Domain-Aware Gaussian Processes [0.0]
  We propose advanced kernel designs that only allow for functions with certain desirable characteristics to be elements of the reproducing kernel Hilbert space (RKHS) We will show the impact of advanced kernel designs on Gaussian processes using several synthetic and two scientific data sets.
  arXiv  Detail & Related papers  (2021-02-05T22:07:56Z)
• UNIPoint: Universally Approximating Point Processes Intensities [125.08205865536577]
  We provide a proof that a class of learnable functions can universally approximate any valid intensity function. We implement UNIPoint, a novel neural point process model, using recurrent neural networks to parameterise sums of basis function upon each event.
  arXiv  Detail & Related papers  (2020-07-28T09:31:56Z)
• Mat\'ern Gaussian processes on Riemannian manifolds [81.15349473870816]
  We show how to generalize the widely-used Mat'ern class of Gaussian processes. We also extend the generalization from the Mat'ern to the widely-used squared exponential process.
  arXiv  Detail & Related papers  (2020-06-17T21:05:42Z)
• SLEIPNIR: Deterministic and Provably Accurate Feature Expansion for Gaussian Process Regression with Derivatives [86.01677297601624]
  We propose a novel approach for scaling GP regression with derivatives based on quadrature Fourier features. We prove deterministic, non-asymptotic and exponentially fast decaying error bounds which apply for both the approximated kernel as well as the approximated posterior.
  arXiv  Detail & Related papers  (2020-03-05T14:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.