Synergistic eigenanalysis of covariance and Hessian matrices for
enhanced binary classification
- URL: http://arxiv.org/abs/2402.09281v1
- Date: Wed, 14 Feb 2024 16:10:42 GMT
- Title: Synergistic eigenanalysis of covariance and Hessian matrices for
enhanced binary classification
- Authors: Agus Hartoyo, Jan Argasi\'nski, Aleksandra Trenk, Kinga Przybylska,
Anna B{\l}asiak, Alessandro Crimi
- Abstract summary: We present a novel approach that combines the eigenanalysis of a covariance matrix evaluated on a training set with a Hessian matrix evaluated on a deep learning model.
Our approach is substantiated by formal proofs that establish its capability to maximize between-class mean distance and minimize within-class variances.
- Score: 75.90957645766676
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Covariance and Hessian matrices have been analyzed separately in the
literature for classification problems. However, integrating these matrices has
the potential to enhance their combined power in improving classification
performance. We present a novel approach that combines the eigenanalysis of a
covariance matrix evaluated on a training set with a Hessian matrix evaluated
on a deep learning model to achieve optimal class separability in binary
classification tasks. Our approach is substantiated by formal proofs that
establish its capability to maximize between-class mean distance and minimize
within-class variances. By projecting data into the combined space of the most
relevant eigendirections from both matrices, we achieve optimal class
separability as per the linear discriminant analysis (LDA) criteria. Empirical
validation across neural and health datasets consistently supports our
theoretical framework and demonstrates that our method outperforms established
methods. Our method stands out by addressing both LDA criteria, unlike PCA and
the Hessian method, which predominantly emphasize one criterion each. This
comprehensive approach captures intricate patterns and relationships, enhancing
classification performance. Furthermore, through the utilization of both LDA
criteria, our method outperforms LDA itself by leveraging higher-dimensional
feature spaces, in accordance with Cover's theorem, which favors linear
separability in higher dimensions. Our method also surpasses kernel-based
methods and manifold learning techniques in performance. Additionally, our
approach sheds light on complex DNN decision-making, rendering them
comprehensible within a 2D space.
Related papers
- Regularized Linear Discriminant Analysis Using a Nonlinear Covariance
Matrix Estimator [11.887333567383239]
Linear discriminant analysis (LDA) is a widely used technique for data classification.
LDA becomes inefficient when the data covariance matrix is ill-conditioned.
Regularized LDA methods have been proposed to cope with such a situation.
arXiv Detail & Related papers (2024-01-31T11:37:14Z) - Random Matrix Analysis to Balance between Supervised and Unsupervised
Learning under the Low Density Separation Assumption [9.620832983703863]
We introduce QLDS, a linear classification model, where the low density separation assumption is implemented via quadratic margin.
We show that particular cases of our algorithm are the least-square support vector machine in the supervised case, the spectral clustering in the fully unsupervised regime, and a class of semi-supervised graph-based approaches.
arXiv Detail & Related papers (2023-10-20T11:46:12Z) - Classification of BCI-EEG based on augmented covariance matrix [0.0]
We propose a new framework based on the augmented covariance extracted from an autoregressive model to improve motor imagery classification.
We will test our approach on several datasets and several subjects using the MOABB framework.
arXiv Detail & Related papers (2023-02-09T09:04:25Z) - Learning Graphical Factor Models with Riemannian Optimization [70.13748170371889]
This paper proposes a flexible algorithmic framework for graph learning under low-rank structural constraints.
The problem is expressed as penalized maximum likelihood estimation of an elliptical distribution.
We leverage geometries of positive definite matrices and positive semi-definite matrices of fixed rank that are well suited to elliptical models.
arXiv Detail & Related papers (2022-10-21T13:19:45Z) - Semi-Supervised Subspace Clustering via Tensor Low-Rank Representation [64.49871502193477]
We propose a novel semi-supervised subspace clustering method, which is able to simultaneously augment the initial supervisory information and construct a discriminative affinity matrix.
Comprehensive experimental results on six commonly-used benchmark datasets demonstrate the superiority of our method over state-of-the-art methods.
arXiv Detail & Related papers (2022-05-21T01:47:17Z) - Self-Weighted Robust LDA for Multiclass Classification with Edge Classes [111.5515086563592]
A novel self-weighted robust LDA with l21-norm based between-class distance criterion, called SWRLDA, is proposed for multi-class classification.
The proposed SWRLDA is easy to implement, and converges fast in practice.
arXiv Detail & Related papers (2020-09-24T12:32:55Z) - Population structure-learned classifier for high-dimension
low-sample-size class-imbalanced problem [3.411873646414169]
Population Structure-learned classifier (PSC) is proposed.
PSC can obtain better generalization performance on IHDLSS.
PSC is superior to the state-of-art methods in IHDLSS.
arXiv Detail & Related papers (2020-09-10T08:33:39Z) - High-Dimensional Quadratic Discriminant Analysis under Spiked Covariance
Model [101.74172837046382]
We propose a novel quadratic classification technique, the parameters of which are chosen such that the fisher-discriminant ratio is maximized.
Numerical simulations show that the proposed classifier not only outperforms the classical R-QDA for both synthetic and real data but also requires lower computational complexity.
arXiv Detail & Related papers (2020-06-25T12:00:26Z) - Sparse Methods for Automatic Relevance Determination [0.0]
We first review automatic relevance determination (ARD) and analytically demonstrate the need to additional regularization or thresholding to achieve sparse models.
We then discuss two classes of methods, regularization based and thresholding based, which build on ARD to learn parsimonious solutions to linear problems.
arXiv Detail & Related papers (2020-05-18T14:08:49Z) - Saliency-based Weighted Multi-label Linear Discriminant Analysis [101.12909759844946]
We propose a new variant of Linear Discriminant Analysis (LDA) to solve multi-label classification tasks.
The proposed method is based on a probabilistic model for defining the weights of individual samples.
The Saliency-based weighted Multi-label LDA approach is shown to lead to performance improvements in various multi-label classification problems.
arXiv Detail & Related papers (2020-04-08T19:40:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.