A Multi-parameter Updating Fourier Online Gradient Descent Algorithm for
Large-scale Nonlinear Classification
- URL: http://arxiv.org/abs/2203.08349v1
- Date: Wed, 16 Mar 2022 02:06:00 GMT
- Title: A Multi-parameter Updating Fourier Online Gradient Descent Algorithm for
Large-scale Nonlinear Classification
- Authors: Yigying Chen
- Abstract summary: The proposed MPU-FOGD is proposed for large-scale nonlinear classification problems based on a novel random feature map.
It is proved that compared with the existing random Fourier feature maps, the proposed random feature map can give a tighter error bound.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Large scale nonlinear classification is a challenging task in the field of
support vector machine. Online random Fourier feature map algorithms are very
important methods for dealing with large scale nonlinear classification
problems. The main shortcomings of these methods are as follows: (1) Since only
the hyperplane vector is updated during learning while the random directions
are fixed, there is no guarantee that these online methods can adapt to the
change of data distribution when the data is coming one by one. (2) The
dimension of the random direction is often higher for obtaining better
classification accuracy, which results in longer test time. In order to
overcome these shortcomings, a multi-parameter updating Fourier online gradient
descent algorithm (MPU-FOGD) is proposed for large-scale nonlinear
classification problems based on a novel random feature map. In the proposed
method, the suggested random feature map has lower dimension while the
multi-parameter updating strategy can guarantee the learning model can better
adapt to the change of data distribution when the data is coming one by one.
Theoretically, it is proved that compared with the existing random Fourier
feature maps, the proposed random feature map can give a tighter error bound.
Empirical studies on several benchmark data sets demonstrate that compared with
the state-of-the-art online random Fourier feature map methods, the proposed
MPU-FOGD can obtain better test accuracy.
Related papers
- Low-rank extended Kalman filtering for online learning of neural
networks from streaming data [71.97861600347959]
We propose an efficient online approximate Bayesian inference algorithm for estimating the parameters of a nonlinear function from a potentially non-stationary data stream.
The method is based on the extended Kalman filter (EKF), but uses a novel low-rank plus diagonal decomposition of the posterior matrix.
In contrast to methods based on variational inference, our method is fully deterministic, and does not require step-size tuning.
arXiv Detail & Related papers (2023-05-31T03:48:49Z) - Online Learning Under A Separable Stochastic Approximation Framework [20.26530917721778]
We propose an online learning algorithm for a class of machine learning models under a separable approximation framework.
We show that the proposed algorithm produces more robust and test performance when compared to other popular learning algorithms.
arXiv Detail & Related papers (2023-05-12T13:53:03Z) - Compound Batch Normalization for Long-tailed Image Classification [77.42829178064807]
We propose a compound batch normalization method based on a Gaussian mixture.
It can model the feature space more comprehensively and reduce the dominance of head classes.
The proposed method outperforms existing methods on long-tailed image classification.
arXiv Detail & Related papers (2022-12-02T07:31:39Z) - Faster Adaptive Federated Learning [84.38913517122619]
Federated learning has attracted increasing attention with the emergence of distributed data.
In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
arXiv Detail & Related papers (2022-12-02T05:07:50Z) - One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive
Least-Squares [8.443742714362521]
We develop an algorithm for one-pass learning which seeks to perfectly fit every new datapoint while changing the parameters in a direction that causes the least change to the predictions on previous datapoints.
Our algorithm uses the memory efficiently by exploiting the structure of the streaming data via an incremental principal component analysis (IPCA)
Our experiments show the effectiveness of the proposed method compared to the baselines.
arXiv Detail & Related papers (2022-07-28T02:01:31Z) - Fast and Robust Online Inference with Stochastic Gradient Descent via
Random Scaling [0.9806910643086042]
We develop a new method of online inference for a vector of parameters estimated by the Polyak-Rtupper averaging procedure of gradient descent algorithms.
Our approach is fully operational with online data and is rigorously underpinned by a functional central limit theorem.
arXiv Detail & Related papers (2021-06-06T15:38:37Z) - Rank-R FNN: A Tensor-Based Learning Model for High-Order Data
Classification [69.26747803963907]
Rank-R Feedforward Neural Network (FNN) is a tensor-based nonlinear learning model that imposes Canonical/Polyadic decomposition on its parameters.
First, it handles inputs as multilinear arrays, bypassing the need for vectorization, and can thus fully exploit the structural information along every data dimension.
We establish the universal approximation and learnability properties of Rank-R FNN, and we validate its performance on real-world hyperspectral datasets.
arXiv Detail & Related papers (2021-04-11T16:37:32Z) - Scalable Marginal Likelihood Estimation for Model Selection in Deep
Learning [78.83598532168256]
Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties.
Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
arXiv Detail & Related papers (2021-04-11T09:50:24Z) - Change Point Detection in Time Series Data using Autoencoders with a
Time-Invariant Representation [69.34035527763916]
Change point detection (CPD) aims to locate abrupt property changes in time series data.
Recent CPD methods demonstrated the potential of using deep learning techniques, but often lack the ability to identify more subtle changes in the autocorrelation statistics of the signal.
We employ an autoencoder-based methodology with a novel loss function, through which the used autoencoders learn a partially time-invariant representation that is tailored for CPD.
arXiv Detail & Related papers (2020-08-21T15:03:21Z) - Online detection of local abrupt changes in high-dimensional Gaussian
graphical models [13.554038901140949]
The problem of identifying change points in high-dimensional Gaussian graphical models (GGMs) in an online fashion is of interest, due to new applications in biology, economics and social sciences.
We develop a novel test to address this problem that is based on the $ell_infty$ norm of the normalized covariance matrix of an appropriately selected portion of incoming data.
arXiv Detail & Related papers (2020-03-16T00:41:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.