Deep Learning with Kernels through RKHM and the Perron-Frobenius Operator

• URL: http://arxiv.org/abs/2305.13588v2
• Date: Sat, 4 Nov 2023 09:25:58 GMT
• Title: Deep Learning with Kernels through RKHM and the Perron-Frobenius Operator
• Authors: Yuka Hashimoto, Masahiro Ikeda, Hachem Kadri
• Abstract summary: Reproducing kernel Hilbert $C*$-module (RKHM) is a generalization of reproducing kernel Hilbert space (RKHS) by means of $C*$-algebra. We derive a new Rademacher generalization bound in this setting and provide a theoretical interpretation of benign overfitting by means of Perron-Frobenius operators.
• Score: 14.877070496733966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reproducing kernel Hilbert $C^*$-module (RKHM) is a generalization of reproducing kernel Hilbert space (RKHS) by means of $C^*$-algebra, and the Perron-Frobenius operator is a linear operator related to the composition of functions. Combining these two concepts, we present deep RKHM, a deep learning framework for kernel methods. We derive a new Rademacher generalization bound in this setting and provide a theoretical interpretation of benign overfitting by means of Perron-Frobenius operators. By virtue of $C^*$-algebra, the dependency of the bound on output dimension is milder than existing bounds. We show that $C^*$-algebra is a suitable tool for deep learning with kernels, enabling us to take advantage of the product structure of operators and to provide a clear connection with convolutional neural networks. Our theoretical analysis provides a new lens through which one can design and analyze deep kernel methods.

Related papers

• Kernel Neural Operators (KNOs) for Scalable, Memory-efficient, Geometrically-flexible Operator Learning [15.050519590538634]
  The Kernel Neural Operator (KNO) is a novel operator learning technique. It uses deep kernel-based integral operators in conjunction with quadrature for function-space approximation of operators. KNOs represent a new paradigm of low-memory, geometrically-flexible, deep operator learning.
  arXiv  Detail & Related papers  (2024-06-30T19:28:12Z)
• Linear quadratic control of nonlinear systems with Koopman operator learning and the Nyström method [16.0198373552099]
  We show how random subspaces can be used to achieve huge computational savings. Our main technical contribution is deriving theoretical guarantees on the effect of the Nystr"om approximation.
  arXiv  Detail & Related papers  (2024-03-05T09:28:40Z)
• Neural Operators with Localized Integral and Differential Kernels [77.76991758980003]
  We present a principled approach to operator learning that can capture local features under two frameworks. We prove that we obtain differential operators under an appropriate scaling of the kernel values of CNNs. To obtain local integral operators, we utilize suitable basis representations for the kernels based on discrete-continuous convolutions.
  arXiv  Detail & Related papers  (2024-02-26T18:59:31Z)
• An approach to the Gaussian RBF kernels via Fock spaces [0.0]
  We use methods from the Fock space and Segal-Bargmann theories to prove several results on the Gaussian RBF kernel. We show how the RBF kernels can be related to some of the most used operators in quantum mechanics and time frequency analysis.
  arXiv  Detail & Related papers  (2022-10-25T16:59:00Z)
• Learning in RKHM: a $C^*$-Algebraic Twist for Kernel Machines [13.23700804428796]
  Supervised learning in reproducing kernel Hilbert space (RKHS) and vector-valued RKHS (vvRKHS) has been investigated for more than 30 years. We provide a new twist by generalizing supervised learning in RKHS and vvRKHS to reproducing kernel Hilbert $C*$-module (RKHM) We show how to construct effective positive-definite kernels by considering the perspective of $C*$-algebra.
  arXiv  Detail & Related papers  (2022-10-21T10:23:54Z)
• Vector-valued Gaussian Processes on Riemannian Manifolds via Gauge Equivariant Projected Kernels [108.60991563944351]
  We present a recipe for constructing gauge equivariant kernels, which induce vector-valued Gaussian processes coherent with geometry. We extend standard Gaussian process training methods, such as variational inference, to this setting.
  arXiv  Detail & Related papers  (2021-10-27T13:31:10Z)
• Neural Operator: Learning Maps Between Function Spaces [75.93843876663128]
  We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We prove a universal approximation theorem for our proposed neural operator, showing that it can approximate any given nonlinear continuous operator. An important application for neural operators is learning surrogate maps for the solution operators of partial differential equations.
  arXiv  Detail & Related papers  (2021-08-19T03:56:49Z)
• Estimating Koopman operators for nonlinear dynamical systems: a nonparametric approach [77.77696851397539]
  The Koopman operator is a mathematical tool that allows for a linear description of non-linear systems. In this paper we capture their core essence as a dual version of the same framework, incorporating them into the Kernel framework. We establish a strong link between kernel methods and Koopman operators, leading to the estimation of the latter through Kernel functions.
  arXiv  Detail & Related papers  (2021-03-25T11:08:26Z)
• On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spaces [208.67848059021915]
  We study the exploration-exploitation tradeoff at the core of reinforcement learning. In particular, we prove that the complexity of the function class $mathcalF$ characterizes the complexity of the function. Our regret bounds are independent of the number of episodes.
  arXiv  Detail & Related papers  (2020-11-09T18:32:22Z)
• Stochastic Flows and Geometric Optimization on the Orthogonal Group [52.50121190744979]
  We present a new class of geometrically-driven optimization algorithms on the orthogonal group $O(d)$. We show that our methods can be applied in various fields of machine learning including deep, convolutional and recurrent neural networks, reinforcement learning, flows and metric learning.
  arXiv  Detail & Related papers  (2020-03-30T15:37:50Z)
• Analysis via Orthonormal Systems in Reproducing Kernel Hilbert $C^*$-Modules and Applications [12.117553807794382]
  We propose a novel data analysis framework with reproducing kernel Hilbert $C*$-module (RKHM) We show the theoretical validity for the construction of orthonormal systems in Hilbert $C*$-modules, and derive concrete procedures for orthonormalization in RKHMs. We apply those to generalize with RKHM kernel principal component analysis and the analysis of dynamical systems with Perron-Frobenius operators.
  arXiv  Detail & Related papers  (2020-03-02T10:01:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.