A new approach to locally adaptive polynomial regression

• URL: http://arxiv.org/abs/2412.19802v2
• Date: Tue, 20 May 2025 14:35:23 GMT
• Title: A new approach to locally adaptive polynomial regression
• Authors: Sabyasachi Chatterjee, Subhajit Goswami, Soumendu Sundar Mukherjee,
• Abstract summary: This paper introduces a new bandwidth selection procedure inspired by the optimality criteria for $ell_$-penalized regression.<n>We obtain non-aptotic risk bounds for the local regression methods based on our bandwidth selection procedure.<n>We show that there is a single ideal choice of a global tuning parameter in each case under which the above-mentioned local adaptivity holds.
• Score: 5.926203312586109
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adaptive bandwidth selection is a fundamental challenge in nonparametric regression. This paper introduces a new bandwidth selection procedure inspired by the optimality criteria for $\ell_0$-penalized regression. Although similar in spirit to Lepski's method and its variants in selecting the largest interval satisfying an admissibility criterion, our approach stems from a distinct philosophy, utilizing criteria based on $\ell_2$-norms of interval projections rather than explicit point and variance estimates. We obtain non-asymptotic risk bounds for the local polynomial regression methods based on our bandwidth selection procedure which adapt (near-)optimally to the local H\"{o}lder exponent of the underlying regression function simultaneously at all points in its domain. Furthermore, we show that there is a single ideal choice of a global tuning parameter in each case under which the above-mentioned local adaptivity holds. The optimal risks of our methods derive from the properties of solutions to a new ``bandwidth selection equation'' which is of independent interest. We believe that the principles underlying our approach provide a new perspective to the classical yet ever relevant problem of locally adaptive nonparametric regression.

Related papers

• Integrated Subset Selection and Bandwidth Estimation Algorithm for Geographically Weighted Regression [0.6642919568083928]
  This study proposes a mathematical-based algorithm for the integrated selection of variable subsets bandwidth estimation in geographically weighted regression.<n>We show that the proposed algorithm provides competitive explanatory power with stable spatially varying patterns, with the ability to select the best subset and account for additional constraints.
  arXiv  Detail & Related papers  (2025-03-21T15:57:59Z)
• GeneralizeFormer: Layer-Adaptive Model Generation across Test-Time Distribution Shifts [58.95913531746308]
  We consider the problem of test-time domain generalization, where a model is trained on several source domains and adjusted on target domains never seen during training. We propose to generate multiple layer parameters on the fly during inference by a lightweight meta-learned transformer, which we call textitGeneralizeFormer.
  arXiv  Detail & Related papers  (2025-02-15T10:10:49Z)
• Robust Local Polynomial Regression with Similarity Kernels [0.0]
  Local Polynomial Regression (LPR) is a widely used nonparametric method for modeling complex relationships. It estimates a regression function by fitting low-degree weights to localized subsets of the data, weighted by proximity. Traditional LPR is sensitive to outliers and high-leverage points, which can significantly affect estimation accuracy. This paper proposes a novel framework that incorporates both predictor and response variables in the weighting mechanism.
  arXiv  Detail & Related papers  (2025-01-18T11:21:26Z)
• Adaptive Conformal Inference by Betting [51.272991377903274]
  We consider the problem of adaptive conformal inference without any assumptions about the data generating process.<n>Existing approaches for adaptive conformal inference are based on optimizing the pinball loss using variants of online gradient descent.<n>We propose a different approach for adaptive conformal inference that leverages parameter-free online convex optimization techniques.
  arXiv  Detail & Related papers  (2024-12-26T18:42:08Z)
• Minmax Trend Filtering: A Locally Adaptive Nonparametric Regression Method via Pointwise Min Max Optimization [4.07926531936425]
  There seems to be no unanimously agreed upon definition of local adaptivity in the literature. We first derive a new pointwise formula for the Fused Lasso estimator in terms of min-max/max-min optimization of penalized local averages. We then propose higher order versions of Fused Lasso which are defined pointwise in terms of min-max/max-min optimization of penalized local regressions.
  arXiv  Detail & Related papers  (2024-10-03T23:15:35Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate. We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Adaptive and non-adaptive minimax rates for weighted Laplacian-eigenmap based nonparametric regression [14.003044924094597]
  We show both adaptive and non-adaptive minimax rates of convergence for a family of weighted Laplacian-Eigenmap based nonparametric regression methods.
  arXiv  Detail & Related papers  (2023-10-31T20:25:36Z)
• Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup [90.26270347459915]
  We propose a novel momentum-based algorithm via utilizing the global descent locally adaptive. textitLADA could greatly reduce the communication rounds and achieves higher accuracy than several baselines.
  arXiv  Detail & Related papers  (2023-07-30T14:53:21Z)
• Sharp Variance-Dependent Bounds in Reinforcement Learning: Best of Both Worlds in Stochastic and Deterministic Environments [48.96971760679639]
  We study variance-dependent regret bounds for Markov decision processes (MDPs) We propose two new environment norms to characterize the fine-grained variance properties of the environment. For model-based methods, we design a variant of the MVP algorithm. In particular, this bound is simultaneously minimax optimal for both and deterministic MDPs.
  arXiv  Detail & Related papers  (2023-01-31T06:54:06Z)
• Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descent [10.108468796986074]
  We study the online statistical inference of model parameters in a contextual bandit framework of decision-making. We propose a general framework for online and adaptive data collection environment that can update decision rules via weighted gradient descent.
  arXiv  Detail & Related papers  (2022-12-30T18:57:08Z)
• Fully Stochastic Trust-Region Sequential Quadratic Programming for Equality-Constrained Optimization Problems [62.83783246648714]
  We propose a sequential quadratic programming algorithm (TR-StoSQP) to solve nonlinear optimization problems with objectives and deterministic equality constraints. The algorithm adaptively selects the trust-region radius and, compared to the existing line-search StoSQP schemes, allows us to utilize indefinite Hessian matrices.
  arXiv  Detail & Related papers  (2022-11-29T05:52:17Z)
• A flexible empirical Bayes approach to multiple linear regression and connections with penalized regression [8.663322701649454]
  We introduce a new empirical Bayes approach for large-scale multiple linear regression. Our approach combines two key ideas: the use of flexible "adaptive shrinkage" priors and variational approximations. We show that the posterior mean from our method solves a penalized regression problem.
  arXiv  Detail & Related papers  (2022-08-23T12:42:57Z)
• Provably tuning the ElasticNet across instances [53.0518090093538]
  We consider the problem of tuning the regularization parameters of Ridge regression, LASSO, and the ElasticNet across multiple problem instances. Our results are the first general learning-theoretic guarantees for this important class of problems.
  arXiv  Detail & Related papers  (2022-07-20T21:22:40Z)
• Benign overfitting and adaptive nonparametric regression [71.70323672531606]
  We construct an estimator which is a continuous function interpolating the data points with high probability. We attain minimax optimal rates under mean squared risk on the scale of H"older classes adaptively to the unknown smoothness.
  arXiv  Detail & Related papers  (2022-06-27T14:50:14Z)
• Off-Policy Evaluation with Policy-Dependent Optimization Response [90.28758112893054]
  We develop a new framework for off-policy evaluation with a textitpolicy-dependent linear optimization response. We construct unbiased estimators for the policy-dependent estimand by a perturbation method. We provide a general algorithm for optimizing causal interventions.
  arXiv  Detail & Related papers  (2022-02-25T20:25:37Z)
• Random Forest Weighted Local Fréchet Regression with Random Objects [18.128663071848923]
  We propose a novel random forest weighted local Fr'echet regression paradigm. Our first method uses these weights as the local average to solve the conditional Fr'echet mean. Second method performs local linear Fr'echet regression, both significantly improving existing Fr'echet regression methods.
  arXiv  Detail & Related papers  (2022-02-10T09:10:59Z)
• Communication-Efficient Distributed Quantile Regression with Optimal Statistical Guarantees [2.064612766965483]
  We address the problem of how to achieve optimal inference in distributed quantile regression without stringent scaling conditions. The difficulties are resolved through a double-smoothing approach that is applied to the local (at each data source) and global objective functions. Despite the reliance on a delicate combination of local and global smoothing parameters, the quantile regression model is fully parametric.
  arXiv  Detail & Related papers  (2021-10-25T17:09:59Z)
• Optimal Rates for Random Order Online Optimization [60.011653053877126]
  We study the citetgarber 2020online, where the loss functions may be chosen by an adversary, but are then presented online in a uniformly random order. We show that citetgarber 2020online algorithms achieve the optimal bounds and significantly improve their stability.
  arXiv  Detail & Related papers  (2021-06-29T09:48:46Z)
• AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods [7.486132958737807]
  We present an adaptive variance reduced method with an implicit approach for adaptivity. We provide convergence guarantees for finite-sum minimization problems and show a faster convergence than SARAH can be achieved if local geometry permits. This algorithm implicitly computes step-size and efficiently estimates local Lipschitz smoothness of functions.
  arXiv  Detail & Related papers  (2021-02-19T01:17:15Z)
• Support estimation in high-dimensional heteroscedastic mean regression [2.28438857884398]
  We consider a linear mean regression model with random design and potentially heteroscedastic, heavy-tailed errors. We use a strictly convex, smooth variant of the Huber loss function with tuning parameter depending on the parameters of the problem. For the resulting estimator we show sign-consistency and optimal rates of convergence in the $ell_infty$ norm.
  arXiv  Detail & Related papers  (2020-11-03T09:46:31Z)
• Online and Distribution-Free Robustness: Regression and Contextual Bandits with Huber Contamination [29.85468294601847]
  We revisit two classic high-dimensional online learning problems, namely linear regression and contextual bandits. We show that our algorithms succeed where conventional methods fail.
  arXiv  Detail & Related papers  (2020-10-08T17:59:05Z)
• Adaptive Online Estimation of Piecewise Polynomial Trends [23.91519151164528]
  We consider the framework of non-stationary optimization with squared error losses and noisy gradient feedback. Motivated from the theory of non-parametric regression, we introduce a new variational constraint. We show that the same policy is minimax optimal for several other non-parametric families of interest.
  arXiv  Detail & Related papers  (2020-09-30T19:30:28Z)
• Adaptive Sampling of Pareto Frontiers with Binary Constraints Using Regression and Classification [0.0]
  We present a novel adaptive optimization algorithm for black-box multi-objective optimization problems with binary constraints. Our method is based on probabilistic regression and classification models, which act as a surrogate for the optimization goals. We also present a novel ellipsoid truncation method to speed up the expected hypervolume calculation.
  arXiv  Detail & Related papers  (2020-08-27T09:15:02Z)
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization [80.03647903934723]
  We prove adaptive gradient methods in expectation of gradient convergence methods. Our analyses shed light on better adaptive gradient methods in optimizing non understanding gradient bounds.
  arXiv  Detail & Related papers  (2018-08-16T20:25:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.