Consistent Interpolating Ensembles via the Manifold-Hilbert Kernel

• URL: http://arxiv.org/abs/2205.09342v1
• Date: Thu, 19 May 2022 06:40:21 GMT
• Title: Consistent Interpolating Ensembles via the Manifold-Hilbert Kernel
• Abstract summary: We devise an ensemble classification method that simultaneously interpolates the training data, and is consistent for a broad class of data distributions. For the sphere, we show that the manifold-Hilbert kernel can be realized as a weighted random partition kernel.
• Score: 11.759162160620678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent research in the theory of overparametrized learning has sought to establish generalization guarantees in the interpolating regime. Such results have been established for a few common classes of methods, but so far not for ensemble methods. We devise an ensemble classification method that simultaneously interpolates the training data, and is consistent for a broad class of data distributions. To this end, we define the manifold-Hilbert kernel for data distributed on a Riemannian manifold. We prove that kernel smoothing regression using the manifold-Hilbert kernel is weakly consistent in the setting of Devroye et al. 1998. For the sphere, we show that the manifold-Hilbert kernel can be realized as a weighted random partition kernel, which arises as an infinite ensemble of partition-based classifiers.

Related papers

• Consistent Geometric Deep Learning via Hilbert Bundles and Cellular Sheaves [68.683840670745]
  We introduce a novel convolutional learning framework for possibly infinite-dimensional signals supported on a manifold.<n>We make HilbNets and, more generally, the convolution operation, implementable via a two-stage sampling procedure.<n>Our results broaden the scope of geometric learning as a whole by lifting classical Laplacian-based frameworks to settings where the signal at each point lives in its own Hilbert space.
  arXiv  Detail & Related papers  (2026-05-07T15:08:58Z)
• A New Kernel Regularity Condition for Distributed Mirror Descent: Broader Coverage and Simpler Analysis [12.17232701440245]
  Hessian relative uniform continuity guarantees convergence under mild conditions.<n>We derive convergence guarantees for mirror descent-based gradient tracking without imposing any restrictive assumptions.
  arXiv  Detail & Related papers  (2026-03-13T09:40:15Z)
• Kernel K-means clustering of distributional data [0.0]
  We consider the problem of clustering a sample of probability distributions from a random distribution on $mathbb Rp$.<n>Our proposed partitioning method makes use of a symmetric, positive-definite kernel $k$ and its associated reproducing kernel Hilbert space $mathcal H$.<n>By mapping each distribution to its corresponding kernel mean embedding in $mathcal H$, we obtain a sample in this RKHS where we carry out the $K$-means clustering procedure.
  arXiv  Detail & Related papers  (2025-09-22T17:11:29Z)
• On the Convergence of Irregular Sampling in Reproducing Kernel Hilbert Spaces [0.0]
  We discuss approximation properties of kernel regression under minimalistic assumptions on both the kernel and the input data. We first prove error estimates in the kernel's RKHS norm. This leads to new results concerning uniform convergence of kernel regression on compact domains.
  arXiv  Detail & Related papers  (2025-04-18T10:57:16Z)
• Learning to Embed Distributions via Maximum Kernel Entropy [0.0]
  Emprimiical data can often be considered as samples from a set of probability distributions. Kernel methods have emerged as a natural approach for learning to classify these distributions. We propose a novel objective for the unsupervised learning of data-dependent distribution kernel.
  arXiv  Detail & Related papers  (2024-08-01T13:34:19Z)
• Wiener Chaos in Kernel Regression: Towards Untangling Aleatoric and Epistemic Uncertainty [0.0]
  We generalize the setting and consider kernel ridge regression with additive i.i.d. nonGaussian measurement noise. We show that our approach allows us to distinguish the uncertainty that stems from the noise in the data samples from the total uncertainty encoded in the GP posterior distribution.
  arXiv  Detail & Related papers  (2023-12-12T16:02:35Z)
• Generative Modeling on Manifolds Through Mixture of Riemannian Diffusion Processes [57.396578974401734]
  We introduce a principled framework for building a generative diffusion process on general manifold. Instead of following the denoising approach of previous diffusion models, we construct a diffusion process using a mixture of bridge processes. We develop a geometric understanding of the mixture process, deriving the drift as a weighted mean of tangent directions to the data points.
  arXiv  Detail & Related papers  (2023-10-11T06:04:40Z)
• Improved learning theory for kernel distribution regression with two-stage sampling [3.154269505086155]
  kernel methods have become a method of choice to tackle the distribution regression problem. We introduce the novel near-unbiased condition on the Hilbertian embeddings, that enables us to provide new error bounds. We show that this near-unbiased condition holds for three important classes of kernels, based on optimal transport and mean embedding.
  arXiv  Detail & Related papers  (2023-08-28T06:29:09Z)
• Variational Autoencoder Kernel Interpretation and Selection for Classification [59.30734371401315]
  This work proposed kernel selection approaches for probabilistic classifiers based on features produced by the convolutional encoder of a variational autoencoder. In the proposed implementation, each latent variable was sampled from the distribution associated with a single kernel of the last encoder's convolution layer, as an individual distribution was created for each kernel. choosing relevant features on the sampled latent variables makes it possible to perform kernel selection, filtering the uninformative features and kernels.
  arXiv  Detail & Related papers  (2022-09-10T17:22:53Z)
• Self-supervised learning with rotation-invariant kernels [4.059849656394191]
  We propose a general kernel framework to design a generic regularization loss that promotes the embedding distribution to be close to the uniform distribution on the hypersphere. Our framework uses rotation-invariant kernels defined on the hypersphere, also known as dot-product kernels. Our experiments demonstrate that using a truncated rotation-invariant kernel provides competitive results compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-07-28T08:06:24Z)
• Recovery Guarantees for Kernel-based Clustering under Non-parametric Mixture Models [26.847612684502998]
  We study the statistical performance of kernel-based clustering algorithms under non-parametric mixture models. We establish a key equivalence between kernel-based data-clustering and kernel density-based clustering.
  arXiv  Detail & Related papers  (2021-10-18T17:23:54Z)
• Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition [54.07797071198249]
  We introduce a new scalable variational Gaussian process approximation which provides a high fidelity approximation while retaining general applicability. We demonstrate that, on a range of regression and classification problems, our approach can exploit input space symmetries such as translations and reflections. Notably, our approach achieves state-of-the-art results on CIFAR-10 among pure GP models.
  arXiv  Detail & Related papers  (2021-06-10T18:17:57Z)
• Kernel learning approaches for summarising and combining posterior similarity matrices [68.8204255655161]
  We build upon the notion of the posterior similarity matrix (PSM) in order to suggest new approaches for summarising the output of MCMC algorithms for Bayesian clustering models. A key contribution of our work is the observation that PSMs are positive semi-definite, and hence can be used to define probabilistically-motivated kernel matrices.
  arXiv  Detail & Related papers  (2020-09-27T14:16:14Z)
• A General Method for Robust Learning from Batches [56.59844655107251]
  We consider a general framework of robust learning from batches, and determine the limits of both classification and distribution estimation over arbitrary, including continuous, domains. We derive the first robust computationally-efficient learning algorithms for piecewise-interval classification, and for piecewise-polynomial, monotone, log-concave, and gaussian-mixture distribution estimation.
  arXiv  Detail & Related papers  (2020-02-25T18:53:25Z)
• Distributed, partially collapsed MCMC for Bayesian Nonparametrics [68.5279360794418]
  We exploit the fact that completely random measures, which commonly used models like the Dirichlet process and the beta-Bernoulli process can be expressed as, are decomposable into independent sub-measures. We use this decomposition to partition the latent measure into a finite measure containing only instantiated components, and an infinite measure containing all other components. The resulting hybrid algorithm can be applied to allow scalable inference without sacrificing convergence guarantees.
  arXiv  Detail & Related papers  (2020-01-15T23:10:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.