Fast and interpretable Support Vector Classification based on the
truncated ANOVA decomposition
- URL: http://arxiv.org/abs/2402.02438v1
- Date: Sun, 4 Feb 2024 10:27:42 GMT
- Title: Fast and interpretable Support Vector Classification based on the
truncated ANOVA decomposition
- Authors: Kseniya Akhalaya, Franziska Nestler, Daniel Potts
- Abstract summary: Support Vector Machines (SVMs) are an important tool for performing classification on scattered data.
We propose solving SVMs in primal form using feature maps based on trigonometric functions or wavelets.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Support Vector Machines (SVMs) are an important tool for performing
classification on scattered data, where one usually has to deal with many data
points in high-dimensional spaces. We propose solving SVMs in primal form using
feature maps based on trigonometric functions or wavelets. In small dimensional
settings the Fast Fourier Transform (FFT) and related methods are a powerful
tool in order to deal with the considered basis functions. For growing
dimensions the classical FFT-based methods become inefficient due to the curse
of dimensionality. Therefore, we restrict ourselves to multivariate basis
functions, each one of them depends only on a small number of dimensions. This
is motivated by the well-known sparsity of effects and recent results regarding
the reconstruction of functions from scattered data in terms of truncated
analysis of variance (ANOVA) decomposition, which makes the resulting model
even interpretable in terms of importance of the features as well as their
couplings. The usage of small superposition dimensions has the consequence that
the computational effort no longer grows exponentially but only polynomially
with respect to the dimension. In order to enforce sparsity regarding the basis
coefficients, we use the frequently applied $\ell_2$-norm and, in addition,
$\ell_1$-norm regularization. The found classifying function, which is the
linear combination of basis functions, and its variance can then be analyzed in
terms of the classical ANOVA decomposition of functions. Based on numerical
examples we show that we are able to recover the signum of a function that
perfectly fits our model assumptions. We obtain better results with
$\ell_1$-norm regularization, both in terms of accuracy and clarity of
interpretability.
Related papers
- Scaling and renormalization in high-dimensional regression [72.59731158970894]
This paper presents a succinct derivation of the training and generalization performance of a variety of high-dimensional ridge regression models.
We provide an introduction and review of recent results on these topics, aimed at readers with backgrounds in physics and deep learning.
arXiv Detail & Related papers (2024-05-01T15:59:00Z) - On the use of the Gram matrix for multivariate functional principal components analysis [0.0]
Dimension reduction is crucial in functional data analysis (FDA)
Existing approaches for functional principal component analysis usually involve the diagonalization of the covariance operator.
We propose to use the inner-product between the curves to estimate the eigenelements of multivariate and multidimensional functional datasets.
arXiv Detail & Related papers (2023-06-22T15:09:41Z) - Kernel-based off-policy estimation without overlap: Instance optimality
beyond semiparametric efficiency [53.90687548731265]
We study optimal procedures for estimating a linear functional based on observational data.
For any convex and symmetric function class $mathcalF$, we derive a non-asymptotic local minimax bound on the mean-squared error.
arXiv Detail & Related papers (2023-01-16T02:57:37Z) - Functional Nonlinear Learning [0.0]
We propose a functional nonlinear learning (FunNoL) method to represent multivariate functional data in a lower-dimensional feature space.
We show that FunNoL provides satisfactory curve classification and reconstruction regardless of data sparsity.
arXiv Detail & Related papers (2022-06-22T23:47:45Z) - Decoupling multivariate functions using a nonparametric filtered tensor
decomposition [0.29360071145551075]
Decoupling techniques aim at providing an alternative representation of the nonlinearity.
The so-called decoupled form is often a more efficient parameterisation of the relationship while being highly structured, favouring interpretability.
In this work two new algorithms, based on filtered tensor decompositions of first order derivative information are introduced.
arXiv Detail & Related papers (2022-05-23T09:34:17Z) - Efficient Multidimensional Functional Data Analysis Using Marginal
Product Basis Systems [2.4554686192257424]
We propose a framework for learning continuous representations from a sample of multidimensional functional data.
We show that the resulting estimation problem can be solved efficiently by the tensor decomposition.
We conclude with a real data application in neuroimaging.
arXiv Detail & Related papers (2021-07-30T16:02:15Z) - Feature Weighted Non-negative Matrix Factorization [92.45013716097753]
We propose the Feature weighted Non-negative Matrix Factorization (FNMF) in this paper.
FNMF learns the weights of features adaptively according to their importances.
It can be solved efficiently with the suggested optimization algorithm.
arXiv Detail & Related papers (2021-03-24T21:17:17Z) - Learning Aggregation Functions [78.47770735205134]
We introduce LAF (Learning Aggregation Functions), a learnable aggregator for sets of arbitrary cardinality.
We report experiments on semi-synthetic and real data showing that LAF outperforms state-of-the-art sum- (max-) decomposition architectures.
arXiv Detail & Related papers (2020-12-15T18:28:53Z) - Piecewise Linear Regression via a Difference of Convex Functions [50.89452535187813]
We present a new piecewise linear regression methodology that utilizes fitting a difference of convex functions (DC functions) to the data.
We empirically validate the method, showing it to be practically implementable, and to have comparable performance to existing regression/classification methods on real-world datasets.
arXiv Detail & Related papers (2020-07-05T18:58:47Z) - Online stochastic gradient descent on non-convex losses from
high-dimensional inference [2.2344764434954256]
gradient descent (SGD) is a popular algorithm for optimization problems in high-dimensional tasks.
In this paper we produce an estimator of non-trivial correlation from data.
We illustrate our approach by applying it to a set of tasks such as phase retrieval, and estimation for generalized models.
arXiv Detail & Related papers (2020-03-23T17:34:06Z) - Invariant Feature Coding using Tensor Product Representation [75.62232699377877]
We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier.
A novel feature model that explicitly consider group action is proposed for principal component analysis and k-means clustering.
arXiv Detail & Related papers (2019-06-05T07:15:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.