Stochastic Gradient Descent for Additive Nonparametric Regression
- URL: http://arxiv.org/abs/2401.00691v2
- Date: Tue, 13 Feb 2024 13:40:53 GMT
- Title: Stochastic Gradient Descent for Additive Nonparametric Regression
- Authors: Xin Chen and Jason M. Klusowski
- Abstract summary: This paper introduces an iterative algorithm for training additive models that enjoys favorable memory storage and computational requirements.
We show that the resulting estimator satisfies an inequality that allows for model mis-specification.
In the well-specified setting, by choosing the learning rate carefully across three distinct stages of training, we demonstrate that its risk is minimax optimal in terms of the dependence on the dimensionality of the data and the size of the training sample.
- Score: 13.28914458950716
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This paper introduces an iterative algorithm for training additive models
that enjoys favorable memory storage and computational requirements. The
algorithm can be viewed as the functional counterpart of stochastic gradient
descent, applied to the coefficients of a truncated basis expansion of the
component functions. We show that the resulting estimator satisfies an oracle
inequality that allows for model mis-specification. In the well-specified
setting, by choosing the learning rate carefully across three distinct stages
of training, we demonstrate that its risk is minimax optimal in terms of the
dependence on the dimensionality of the data and the size of the training
sample. We further illustrate the computational benefits by comparing the
approach with traditional backfitting on two real-world datasets.
Related papers
- Nonparametric Linear Feature Learning in Regression Through
Regularisation [0.0]
This study focuses on supervised learning scenarios where the information resides within a lower-dimensional linear subspace of the data.
We propose a novel method for linear feature learning with non-parametric prediction, which simultaneously estimates the linear subspace.
Our approach employs empirical risk minimisation augmented with a penalty on derivatives, ensuring versatility.
arXiv Detail & Related papers (2023-07-24T12:52:55Z) - Generalizing Backpropagation for Gradient-Based Interpretability [103.2998254573497]
We show that the gradient of a model is a special case of a more general formulation using semirings.
This observation allows us to generalize the backpropagation algorithm to efficiently compute other interpretable statistics.
arXiv Detail & Related papers (2023-07-06T15:19:53Z) - Learning Unnormalized Statistical Models via Compositional Optimization [73.30514599338407]
Noise-contrastive estimation(NCE) has been proposed by formulating the objective as the logistic loss of the real data and the artificial noise.
In this paper, we study it a direct approach for optimizing the negative log-likelihood of unnormalized models.
arXiv Detail & Related papers (2023-06-13T01:18:16Z) - Refining Amortized Posterior Approximations using Gradient-Based Summary
Statistics [0.9176056742068814]
We present an iterative framework to improve the amortized approximations of posterior distributions in the context of inverse problems.
We validate our method in a controlled setting by applying it to a stylized problem, and observe improved posterior approximations with each iteration.
arXiv Detail & Related papers (2023-05-15T15:47:19Z) - Semiparametric Regression for Spatial Data via Deep Learning [17.63607438860882]
We use a sparsely connected deep neural network with rectified linear unit (ReLU) activation function to estimate the unknown regression function.
Our method can handle well large data set owing to the gradient descent optimization algorithm.
arXiv Detail & Related papers (2023-01-10T01:55:55Z) - Adaptive LASSO estimation for functional hidden dynamic geostatistical
model [69.10717733870575]
We propose a novel model selection algorithm based on a penalized maximum likelihood estimator (PMLE) for functional hiddenstatistical models (f-HD)
The algorithm is based on iterative optimisation and uses an adaptive least absolute shrinkage and selector operator (GMSOLAS) penalty function, wherein the weights are obtained by the unpenalised f-HD maximum-likelihood estimators.
arXiv Detail & Related papers (2022-08-10T19:17:45Z) - Feature space approximation for kernel-based supervised learning [2.653409741248232]
The goal is to reduce the size of the training data, resulting in lower storage consumption and computational complexity.
We demonstrate significant improvements in comparison to the computation of data-driven predictions involving the full training data set.
The method is applied to classification and regression problems from different application areas such as image recognition, system identification, and oceanographic time series analysis.
arXiv Detail & Related papers (2020-11-25T11:23:58Z) - A Nonconvex Framework for Structured Dynamic Covariance Recovery [24.471814126358556]
We propose a flexible yet interpretable model for high-dimensional data with time-varying second order statistics.
Motivated by the literature, we quantify factorization and smooth temporal data.
We show that our approach outperforms existing baselines.
arXiv Detail & Related papers (2020-11-11T07:09:44Z) - Slice Sampling for General Completely Random Measures [74.24975039689893]
We present a novel Markov chain Monte Carlo algorithm for posterior inference that adaptively sets the truncation level using auxiliary slice variables.
The efficacy of the proposed algorithm is evaluated on several popular nonparametric models.
arXiv Detail & Related papers (2020-06-24T17:53:53Z) - Deep Dimension Reduction for Supervised Representation Learning [51.10448064423656]
We propose a deep dimension reduction approach to learning representations with essential characteristics.
The proposed approach is a nonparametric generalization of the sufficient dimension reduction method.
We show that the estimated deep nonparametric representation is consistent in the sense that its excess risk converges to zero.
arXiv Detail & Related papers (2020-06-10T14:47:43Z) - Asymptotic Analysis of an Ensemble of Randomly Projected Linear
Discriminants [94.46276668068327]
In [1], an ensemble of randomly projected linear discriminants is used to classify datasets.
We develop a consistent estimator of the misclassification probability as an alternative to the computationally-costly cross-validation estimator.
We also demonstrate the use of our estimator for tuning the projection dimension on both real and synthetic data.
arXiv Detail & Related papers (2020-04-17T12:47:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.