Importance Sampling for Nonlinear Models

• URL: http://arxiv.org/abs/2505.12353v1
• Date: Sun, 18 May 2025 10:34:39 GMT
• Title: Importance Sampling for Nonlinear Models
• Authors: Prakash Palanivelu Rajmohan, Fred Roosta,
• Abstract summary: We introduce the concept of the adjoint operator of a nonlinear map.<n>We demonstrate that sampling based on these notions of norm and leverage scores provides approximation guarantees for the underlying nonlinear mapping.
• Score: 5.421981644827842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While norm-based and leverage-score-based methods have been extensively studied for identifying "important" data points in linear models, analogous tools for nonlinear models remain significantly underdeveloped. By introducing the concept of the adjoint operator of a nonlinear map, we address this gap and generalize norm-based and leverage-score-based importance sampling to nonlinear settings. We demonstrate that sampling based on these generalized notions of norm and leverage scores provides approximation guarantees for the underlying nonlinear mapping, similar to linear subspace embeddings. As direct applications, these nonlinear scores not only reduce the computational complexity of training nonlinear models by enabling efficient sampling over large datasets but also offer a novel mechanism for model explainability and outlier detection. Our contributions are supported by both theoretical analyses and experimental results across a variety of supervised learning scenarios.

Related papers

• Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models [51.85815025140659]
  Modern Machine Learning (ML) and Deep Neural Networks (DNNs) often operate on high-dimensional data.<n>In particular, the proportional regime where the data dimension, sample size, and number of model parameters are all large gives rise to novel and sometimes counterintuitive behaviors.<n>This paper extends traditional Random Matrix Theory (RMT) beyond eigenvalue-based analysis of linear models to address the challenges posed by nonlinear ML models.
  arXiv  Detail & Related papers  (2025-06-16T06:54:08Z)
• TD(0) Learning converges for Polynomial mixing and non-linear functions [49.1574468325115]
  We present theoretical findings for TD learning under more applicable assumptions.<n>This is the first proof of TD(0) convergence on Markov data under universal and-independent step sizes.<n>Our results include bounds for linear models and non-linear under generalized gradients and H"older continuity.
  arXiv  Detail & Related papers  (2025-02-08T22:01:02Z)
• Mixture of partially linear experts [0.0]
  We propose a partially linear structure that incorporates unspecified functions to capture nonlinear relationships. We establish the identifiability of the proposed model under mild conditions and introduce a practical estimation algorithm.
  arXiv  Detail & Related papers  (2024-05-05T12:10:37Z)
• Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regression [0.0]
  This study explores the predictive properties of over parameterized nonlinear regression within the Bayesian framework. Posterior contraction is established for generalized linear and single-neuron models with Lipschitz continuous activation functions. The proposed method was validated via numerical simulations and a real data application.
  arXiv  Detail & Related papers  (2024-04-06T04:22:48Z)
• Learning Linear Causal Representations from Interventions under General Nonlinear Mixing [52.66151568785088]
  We prove strong identifiability results given unknown single-node interventions without access to the intervention targets. This is the first instance of causal identifiability from non-paired interventions for deep neural network embeddings.
  arXiv  Detail & Related papers  (2023-06-04T02:32:12Z)
• On the Detection and Quantification of Nonlinearity via Statistics of the Gradients of a Black-Box Model [0.0]
  Detection and identification of nonlinearity is a task of high importance for structural dynamics. A method to detect nonlinearity is proposed, based on the distribution of the gradients of a data-driven model.
  arXiv  Detail & Related papers  (2023-02-15T23:15:22Z)
• Towards a mathematical understanding of learning from few examples with nonlinear feature maps [68.8204255655161]
  We consider the problem of data classification where the training set consists of just a few data points. We reveal key relationships between the geometry of an AI model's feature space, the structure of the underlying data distributions, and the model's generalisation capabilities.
  arXiv  Detail & Related papers  (2022-11-07T14:52:58Z)
• Learning from few examples with nonlinear feature maps [68.8204255655161]
  We explore the phenomenon and reveal key relationships between dimensionality of AI model's feature space, non-degeneracy of data distributions, and the model's generalisation capabilities. The main thrust of our present analysis is on the influence of nonlinear feature transformations mapping original data into higher- and possibly infinite-dimensional spaces on the resulting model's generalisation capabilities.
  arXiv  Detail & Related papers  (2022-03-31T10:36:50Z)
• Lower Bounds on the Generalization Error of Nonlinear Learning Models [2.1030878979833467]
  We study in this paper lower bounds for the generalization error of models derived from multi-layer neural networks, in the regime where the size of the layers is commensurate with the number of samples in the training data. We show that unbiased estimators have unacceptable performance for such nonlinear networks in this regime. We derive explicit generalization lower bounds for general biased estimators, in the cases of linear regression and of two-layered networks.
  arXiv  Detail & Related papers  (2021-03-26T20:37:54Z)
• Hessian Eigenspectra of More Realistic Nonlinear Models [73.31363313577941]
  We make a emphprecise characterization of the Hessian eigenspectra for a broad family of nonlinear models. Our analysis takes a step forward to identify the origin of many striking features observed in more complex machine learning models.
  arXiv  Detail & Related papers  (2021-03-02T06:59:52Z)
• Learning Fast Approximations of Sparse Nonlinear Regression [50.00693981886832]
  In this work, we bridge the gap by introducing the Threshold Learned Iterative Shrinkage Algorithming (NLISTA) Experiments on synthetic data corroborate our theoretical results and show our method outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-10-26T11:31:08Z)
• Inject Machine Learning into Significance Test for Misspecified Linear Models [14.672773981251574]
  We present a simple and effective assumption-free method for linear approximation in both linear and non-linear scenarios. Experimental results show that our estimator significantly outperforms linear regression for non-linear ground truth functions.
  arXiv  Detail & Related papers  (2020-06-04T23:22:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.