Bagged Polynomial Regression and Neural Networks

• URL: http://arxiv.org/abs/2205.08609v2
• Date: Tue, 17 Sep 2024 21:32:14 GMT
• Title: Bagged Polynomial Regression and Neural Networks
• Authors: Sylvia Klosin, Jaume Vives-i-Bastida,
• Abstract summary: Series and dataset regression are able to approximate the same function classes as neural networks. textitbagged regression (BPR) is an attractive alternative to neural networks. BPR performs as well as neural networks in crop classification using satellite data.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Series and polynomial regression are able to approximate the same function classes as neural networks. However, these methods are rarely used in practice, although they offer more interpretability than neural networks. In this paper, we show that a potential reason for this is the slow convergence rate of polynomial regression estimators and propose the use of \textit{bagged} polynomial regression (BPR) as an attractive alternative to neural networks. Theoretically, we derive new finite sample and asymptotic $L^2$ convergence rates for series estimators. We show that the rates can be improved in smooth settings by splitting the feature space and generating polynomial features separately for each partition. Empirically, we show that our proposed estimator, the BPR, can perform as well as more complex models with more parameters. Our estimator also performs close to state-of-the-art prediction methods in the benchmark MNIST handwritten digit dataset. We demonstrate that BPR performs as well as neural networks in crop classification using satellite data, a setting where prediction accuracy is critical and interpretability is often required for addressing research questions.

Related papers

• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Nonparametric regression using over-parameterized shallow ReLU neural networks [10.339057554827392]
  We show that neural networks can achieve minimax optimal rates of convergence (up to logarithmic factors) for learning functions from certain smooth function classes. It is assumed that the regression function is from the H"older space with smoothness $alpha(d+3)/2$ or a variation space corresponding to shallow neural networks. As a byproduct, we derive a new size-independent bound for the local Rademacher complexity of shallow ReLU neural networks.
  arXiv  Detail & Related papers  (2023-06-14T07:42:37Z)
• Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression [12.21422686958087]
  We study the approximation capacity of some variation spaces corresponding to shallow ReLU$k$ neural networks. For functions with less smoothness, the approximation rates in terms of the variation norm are established. We show that shallow neural networks can achieve the minimax optimal rates for learning H"older functions.
  arXiv  Detail & Related papers  (2023-04-04T06:35:02Z)
• A predictive physics-aware hybrid reduced order model for reacting flows [65.73506571113623]
  A new hybrid predictive Reduced Order Model (ROM) is proposed to solve reacting flow problems. The number of degrees of freedom is reduced from thousands of temporal points to a few POD modes with their corresponding temporal coefficients. Two different deep learning architectures have been tested to predict the temporal coefficients.
  arXiv  Detail & Related papers  (2023-01-24T08:39:20Z)
• Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets [57.06026574261203]
  We provide theoretical evidence for learning low-dimensional state spaces, which can also model long-term memory. Experiments corroborate our theory, demonstrating extrapolation via learning low-dimensional state spaces with both linear and non-linear RNNs.
  arXiv  Detail & Related papers  (2022-10-25T14:45:15Z)
• Towards an Understanding of Benign Overfitting in Neural Networks [104.2956323934544]
  Modern machine learning models often employ a huge number of parameters and are typically optimized to have zero training loss. We examine how these benign overfitting phenomena occur in a two-layer neural network setting. We show that it is possible for the two-layer ReLU network interpolator to achieve a near minimax-optimal learning rate.
  arXiv  Detail & Related papers  (2021-06-06T19:08:53Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)
• Towards a mathematical framework to inform Neural Network modelling via Polynomial Regression [0.0]
  It is shown that almost identical predictions can be made when certain conditions are met locally. When learning from generated data, the proposed method producess that approximate correctly the data locally.
  arXiv  Detail & Related papers  (2021-02-07T17:56:16Z)
• Sparsely constrained neural networks for model discovery of PDEs [0.0]
  We present a modular framework that determines the sparsity pattern of a deep-learning based surrogate using any sparse regression technique. We show how a different network architecture and sparsity estimator improve model discovery accuracy and convergence on several benchmark examples.
  arXiv  Detail & Related papers  (2020-11-09T11:02:40Z)
• Measurement error models: from nonparametric methods to deep neural networks [3.1798318618973362]
  We propose an efficient neural network design for estimating measurement error models. We use a fully connected feed-forward neural network to approximate the regression function $f(x)$. We conduct an extensive numerical study to compare the neural network approach with classical nonparametric methods.
  arXiv  Detail & Related papers  (2020-07-15T06:05:37Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.