Asymptotic Bounds for Smoothness Parameter Estimates in Gaussian Process
Interpolation
- URL: http://arxiv.org/abs/2203.05400v5
- Date: Mon, 27 Nov 2023 16:08:49 GMT
- Title: Asymptotic Bounds for Smoothness Parameter Estimates in Gaussian Process
Interpolation
- Authors: Toni Karvonen
- Abstract summary: The smoothness of a Mat'ern kernel determines many important properties of the model in the large data limit.
We prove that the maximum likelihood estimate of the smoothness parameter cannot be understated under the truth.
We show that maximum likelihood estimation recovers the true smoothness for a class of compactly supported self-similar functions.
- Score: 3.8979646385036175
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: It is common to model a deterministic response function, such as the output
of a computer experiment, as a Gaussian process with a Mat\'ern covariance
kernel. The smoothness parameter of a Mat\'ern kernel determines many important
properties of the model in the large data limit, including the rate of
convergence of the conditional mean to the response function. We prove that the
maximum likelihood estimate of the smoothness parameter cannot asymptotically
undersmooth the truth when the data are obtained on a fixed bounded subset of
$\mathbb{R}^d$. That is, if the data-generating response function has Sobolev
smoothness $\nu_0 > d/2$, then the smoothness parameter estimate cannot be
asymptotically less than $\nu_0$. The lower bound is sharp. Additionally, we
show that maximum likelihood estimation recovers the true smoothness for a
class of compactly supported self-similar functions. For cross-validation we
prove an asymptotic lower bound $\nu_0 - d/2$, which however is unlikely to be
sharp. The results are based on approximation theory in Sobolev spaces and some
general theorems that restrict the set of values that the parameter estimators
can take.
Related papers
- Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods [25.831462008050387]
The Gradient Descent (SGD) algorithm has triggered people's interest due to its good performance in practice but lack of theoretical understanding.
It still remains unclear whether the lastiterate convergence can be provably extended to wider composite optimization and non-Euclidean norms.
arXiv Detail & Related papers (2023-12-13T21:41:06Z) - On Computationally Efficient Learning of Exponential Family
Distributions [33.229944519289795]
We focus on the setting where the support as well as the natural parameters are appropriately bounded.
Our method achives the order-optimal sample complexity of $O(sf log(k)/alpha2)$ when tailored for node-wise-sparse random fields.
arXiv Detail & Related papers (2023-09-12T17:25:32Z) - Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels [78.6096486885658]
We introduce lower bounds to the linearized Laplace approximation of the marginal likelihood.
These bounds are amenable togradient-based optimization and allow to trade off estimation accuracy against computational complexity.
arXiv Detail & Related papers (2023-06-06T19:02:57Z) - Efficient displacement convex optimization with particle gradient
descent [57.88860627977882]
Particle gradient descent is widely used to optimize functions of probability measures.
This paper considers particle gradient descent with a finite number of particles and establishes its theoretical guarantees to optimize functions that are emphdisplacement convex in measures.
arXiv Detail & Related papers (2023-02-09T16:35:59Z) - Kernel-based off-policy estimation without overlap: Instance optimality
beyond semiparametric efficiency [53.90687548731265]
We study optimal procedures for estimating a linear functional based on observational data.
For any convex and symmetric function class $mathcalF$, we derive a non-asymptotic local minimax bound on the mean-squared error.
arXiv Detail & Related papers (2023-01-16T02:57:37Z) - Stochastic Inexact Augmented Lagrangian Method for Nonconvex Expectation
Constrained Optimization [88.0031283949404]
Many real-world problems have complicated non functional constraints and use a large number of data points.
Our proposed method outperforms an existing method with the previously best-known result.
arXiv Detail & Related papers (2022-12-19T14:48:54Z) - Randomized Coordinate Subgradient Method for Nonsmooth Composite
Optimization [11.017632675093628]
Coordinate-type subgradient methods for addressing nonsmooth problems are relatively underexplored due to the set of properties of the Lipschitz-type assumption.
arXiv Detail & Related papers (2022-06-30T02:17:11Z) - Reducing the Variance of Gaussian Process Hyperparameter Optimization
with Preconditioning [54.01682318834995]
Preconditioning is a highly effective step for any iterative method involving matrix-vector multiplication.
We prove that preconditioning has an additional benefit that has been previously unexplored.
It simultaneously can reduce variance at essentially negligible cost.
arXiv Detail & Related papers (2021-07-01T06:43:11Z) - Finding Global Minima via Kernel Approximations [90.42048080064849]
We consider the global minimization of smooth functions based solely on function evaluations.
In this paper, we consider an approach that jointly models the function to approximate and finds a global minimum.
arXiv Detail & Related papers (2020-12-22T12:59:30Z) - Tight Nonparametric Convergence Rates for Stochastic Gradient Descent
under the Noiseless Linear Model [0.0]
We analyze the convergence of single-pass, fixed step-size gradient descent on the least-square risk under this model.
As a special case, we analyze an online algorithm for estimating a real function on the unit interval from the noiseless observation of its value at randomly sampled points.
arXiv Detail & Related papers (2020-06-15T08:25:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.