An Upper Bound of the Bias of Nadaraya-Watson Kernel Regression under Lipschitz Assumptions

• URL: http://arxiv.org/abs/2001.10972v2
• Date: Thu, 30 Jan 2020 09:01:12 GMT
• Title: An Upper Bound of the Bias of Nadaraya-Watson Kernel Regression under Lipschitz Assumptions
• Abstract summary: We propose an upper bound of the Nadaraya-Watson kernel estimator's bias which holds for finite bandwidths. We believe that this work has potential applications in those fields where some hard guarantees on the error are needed.
• Score: 37.02885044489595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Nadaraya-Watson kernel estimator is among the most popular nonparameteric regression technique thanks to its simplicity. Its asymptotic bias has been studied by Rosenblatt in 1969 and has been reported in a number of related literature. However, Rosenblatt's analysis is only valid for infinitesimal bandwidth. In contrast, we propose in this paper an upper bound of the bias which holds for finite bandwidths. Moreover, contrarily to the classic analysis we allow for discontinuous first order derivative of the regression function, we extend our bounds for multidimensional domains and we include the knowledge of the bound of the regression function when it exists and if it is known, to obtain a tighter bound. We believe that this work has potential applications in those fields where some hard guarantees on the error are needed

Related papers

• Lasso Penalization for High-Dimensional Beta Regression Models: Computation, Analysis, and Inference [3.330229314824914]
  We develop a framework for non-asymptotic predictors with a negative log-likelihood function.<n>A gradient is devised for optimizing the resulting penalized negative log-likelihood function.<n>Our theoretical analysis is corroborated via simulation, and a real data example concerning the prediction of county-level incarceration is presented.
  arXiv  Detail & Related papers  (2025-07-26T23:19:17Z)
• Limit Theorems for Stochastic Gradient Descent with Infinite Variance [51.4853131023238]
  We show that the gradient descent algorithm can be characterized as the stationary distribution of a suitably defined Ornstein-rnstein process driven by an appropriate L'evy process.<n>We also explore the applications of these results in linear regression and logistic regression models.
  arXiv  Detail & Related papers  (2024-10-21T09:39:10Z)
• On the Convergence of Gradient Descent for Large Learning Rates [55.33626480243135]
  We show that convergence is impossible when a fixed step size is used. We provide a proof of this in the case of linear neural networks with a squared loss. We also prove the impossibility of convergence for more general losses without requiring strong assumptions such as Lipschitz continuity for the gradient.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• A Bound on the Maximal Marginal Degrees of Freedom [0.0]
  Common kernel ridge regression is expensive in memory allocation and computation time. This paper addresses low rank approximations and surrogates for kernel ridge regression.
  arXiv  Detail & Related papers  (2024-02-20T10:25:44Z)
• Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum [6.749750044497731]
  We prove the phenomena of tempered overfitting and catastrophic overfitting under the sub-Gaussian design assumption. We also identify that the independence of the features plays an important role in guaranteeing tempered overfitting.
  arXiv  Detail & Related papers  (2024-02-02T10:36:53Z)
• Curvature-Independent Last-Iterate Convergence for Games on Riemannian Manifolds [77.4346324549323]
  We show that a step size agnostic to the curvature of the manifold achieves a curvature-independent and linear last-iterate convergence rate. To the best of our knowledge, the possibility of curvature-independent rates and/or last-iterate convergence has not been considered before.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• Kernel-based off-policy estimation without overlap: Instance optimality beyond semiparametric efficiency [53.90687548731265]
  We study optimal procedures for estimating a linear functional based on observational data. For any convex and symmetric function class $mathcalF$, we derive a non-asymptotic local minimax bound on the mean-squared error.
  arXiv  Detail & Related papers  (2023-01-16T02:57:37Z)
• Learning linear operators: Infinite-dimensional regression as a well-behaved non-compact inverse problem [4.503368323711748]
  We consider the problem of learning a linear operator $theta$ between two Hilbert spaces from empirical observations. We show that this goal can be reformulated as an inverse problem for $theta$ with the feature that its forward operator is generally non-compact. We prove that this inverse problem is equivalent to the known compact inverse problem associated with scalar response regression derivation.
  arXiv  Detail & Related papers  (2022-11-16T12:33:01Z)
• Instance-Dependent Generalization Bounds via Optimal Transport [51.71650746285469]
  Existing generalization bounds fail to explain crucial factors that drive the generalization of modern neural networks. We derive instance-dependent generalization bounds that depend on the local Lipschitz regularity of the learned prediction function in the data space. We empirically analyze our generalization bounds for neural networks, showing that the bound values are meaningful and capture the effect of popular regularization methods during training.
  arXiv  Detail & Related papers  (2022-11-02T16:39:42Z)
• Online nonparametric regression with Sobolev kernels [99.12817345416846]
  We derive the regret upper bounds on the classes of Sobolev spaces $W_pbeta(mathcalX)$, $pgeq 2, beta>fracdp$. The upper bounds are supported by the minimax regret analysis, which reveals that in the cases $beta> fracd2$ or $p=infty$ these rates are (essentially) optimal.
  arXiv  Detail & Related papers  (2021-02-06T15:05:14Z)
• Early stopping and polynomial smoothing in regression with reproducing kernels [2.0411082897313984]
  We study the problem of early stopping for iterative learning algorithms in a reproducing kernel Hilbert space (RKHS) We present a data-driven rule to perform early stopping without a validation set that is based on the so-called minimum discrepancy principle. The proposed rule is proved to be minimax-optimal over different types of kernel spaces.
  arXiv  Detail & Related papers  (2020-07-14T05:27:18Z)
• Minimax Semiparametric Learning With Approximate Sparsity [3.5136198842746524]
  This paper formalizes the concept of approximate model sparsity through classical semi-parametric theory.<n>We derive minimax rates for a regression slope and an average derivative, finding these bounds to be substantially larger than those in low-dimensional, semi-parametric settings.
  arXiv  Detail & Related papers  (2019-12-27T16:13:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.