Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates

• URL: http://arxiv.org/abs/2107.05598v1
• Date: Mon, 12 Jul 2021 17:29:08 GMT
• Title: Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates
• Authors: Johannes J. Brust
• Abstract summary: We exploit the property that the number of model parameters typically exceeds the data in one batch to compute search directions. We develop two algorithms that estimate Jacobian matrices and perform well when compared to state-of-the-art methods.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For large nonlinear least squares loss functions in machine learning we exploit the property that the number of model parameters typically exceeds the data in one batch. This implies a low-rank structure in the Hessian of the loss, which enables effective means to compute search directions. Using this property, we develop two algorithms that estimate Jacobian matrices and perform well when compared to state-of-the-art methods.

Related papers

• Spectral Entry-wise Matrix Estimation for Low-Rank Reinforcement Learning [53.445068584013896]
  We study matrix estimation problems arising in reinforcement learning (RL) with low-rank structure. In low-rank bandits, the matrix to be recovered specifies the expected arm rewards, and for low-rank Markov Decision Processes (MDPs), it may for example characterize the transition kernel of the MDP. We show that simple spectral-based matrix estimation approaches efficiently recover the singular subspaces of the matrix and exhibit nearly-minimal entry-wise error.
  arXiv  Detail & Related papers  (2023-10-10T17:06:41Z)
• Efficient Learning of Minimax Risk Classifiers in High Dimensions [3.093890460224435]
  High-dimensional data is common in multiple areas, such as health care and genomics, where the number of features can be tens of thousands. In this paper, we leverage such methods to obtain an efficient learning algorithm for the recently proposed minimax risk classifiers. Experiments on multiple high-dimensional datasets show that the proposed algorithm is efficient in high-dimensional scenarios.
  arXiv  Detail & Related papers  (2023-06-11T11:08:20Z)
• An evaluation framework for dimensionality reduction through sectional curvature [59.40521061783166]
  In this work, we aim to introduce the first highly non-supervised dimensionality reduction performance metric. To test its feasibility, this metric has been used to evaluate the performance of the most commonly used dimension reduction algorithms. A new parameterized problem instance generator has been constructed in the form of a function generator.
  arXiv  Detail & Related papers  (2023-03-17T11:59:33Z)
• An Efficient Method for Sample Adversarial Perturbations against Nonlinear Support Vector Machines [8.000799046379749]
  We investigate the sample adversarial perturbations for nonlinear support vector machines (SVMs) Due to the implicit form of the nonlinear functions mapping data to the feature space, it is difficult to obtain the explicit form of the adversarial perturbations. By exploring the special property of nonlinear SVMs, we transform the optimization problem of attacking nonlinear SVMs into a nonlinear KKT system.
  arXiv  Detail & Related papers  (2022-06-12T05:21:51Z)
• Implicit Parameter-free Online Learning with Truncated Linear Models [51.71216912089413]
  parameter-free algorithms are online learning algorithms that do not require setting learning rates. We propose new parameter-free algorithms that can take advantage of truncated linear models through a new update that has an "implicit" flavor. Based on a novel decomposition of the regret, the new update is efficient, requires only one gradient at each step, never overshoots the minimum of the truncated model, and retains the favorable parameter-free properties.
  arXiv  Detail & Related papers  (2022-03-19T13:39:49Z)
• Sparse PCA via $l_{2,p}$-Norm Regularization for Unsupervised Feature Selection [138.97647716793333]
  We propose a simple and efficient unsupervised feature selection method, by combining reconstruction error with $l_2,p$-norm regularization. We present an efficient optimization algorithm to solve the proposed unsupervised model, and analyse the convergence and computational complexity of the algorithm theoretically.
  arXiv  Detail & Related papers  (2020-12-29T04:08:38Z)
• Binary Matrix Factorisation via Column Generation [6.445605125467574]
  We consider the problem of low-rank binary matrix factorisation (BMF) under arithmetic. We formulate the problem as a mixed integer linear program and use a large scale optimisation technique of column generation to solve it. Experimental results on real world datasets demonstrate that our proposed method is effective at producing highly accurate factorisations.
  arXiv  Detail & Related papers  (2020-11-09T14:27:36Z)
• Efficient Optimization Methods for Extreme Similarity Learning with Nonlinear Embeddings [12.119973339250848]
  We study the problem of learning similarity by using nonlinear embedding models from all possible pairs. This paper aims to extend results for general nonlinear embeddings.
  arXiv  Detail & Related papers  (2020-10-26T12:09:09Z)
• Progressive Batching for Efficient Non-linear Least Squares [31.082253632197023]
  Most improvements of the basic Gauss-Newton tackle convergence guarantees or leverage the sparsity of the underlying problem structure for computational speedup. Our work borrows ideas from both machine learning and statistics, and we present an approach for non-linear least-squares that guarantees convergence while at the same time significantly reduces the required amount of computation.
  arXiv  Detail & Related papers  (2020-10-21T13:00:04Z)
• Learning Mixtures of Low-Rank Models [89.39877968115833]
  We study the problem of learning computational mixtures of low-rank models. We develop an algorithm that is guaranteed to recover the unknown matrices with near-optimal sample. In addition, the proposed algorithm is provably stable against random noise.
  arXiv  Detail & Related papers  (2020-09-23T17:53:48Z)
• A Survey on Large-scale Machine Learning [67.6997613600942]
  Machine learning can provide deep insights into data, allowing machines to make high-quality predictions. Most sophisticated machine learning approaches suffer from huge time costs when operating on large-scale data. Large-scale Machine Learning aims to learn patterns from big data with comparable performance efficiently.
  arXiv  Detail & Related papers  (2020-08-10T06:07:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.