A Doubly Regularized Linear Discriminant Analysis Classifier with Automatic Parameter Selection

• URL: http://arxiv.org/abs/2004.13335v2
• Date: Sat, 27 Mar 2021 17:44:19 GMT
• Title: A Doubly Regularized Linear Discriminant Analysis Classifier with Automatic Parameter Selection
• Authors: Alam Zaib, Tarig Ballal, Shahid Khattak and Tareq Y. Al-Naffouri
• Abstract summary: Linear discriminant analysis (LDA) based classifiers tend to falter in many practical settings where the training data size is smaller than, or comparable to, the number of features. We propose a doubly regularized LDA classifier that we denote as R2LDA. Results obtained from both synthetic and real data demonstrate the consistency and effectiveness of the proposed R2LDA approach.
• Score: 24.027886914804775
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Linear discriminant analysis (LDA) based classifiers tend to falter in many practical settings where the training data size is smaller than, or comparable to, the number of features. As a remedy, different regularized LDA (RLDA) methods have been proposed. These methods may still perform poorly depending on the size and quality of the available training data. In particular, the test data deviation from the training data model, for example, due to noise contamination, can cause severe performance degradation. Moreover, these methods commit further to the Gaussian assumption (upon which LDA is established) to tune their regularization parameters, which may compromise accuracy when dealing with real data. To address these issues, we propose a doubly regularized LDA classifier that we denote as R2LDA. In the proposed R2LDA approach, the RLDA score function is converted into an inner product of two vectors. By substituting the expressions of the regularized estimators of these vectors, we obtain the R2LDA score function that involves two regularization parameters. To set the values of these parameters, we adopt three existing regularization techniques; the constrained perturbation regularization approach (COPRA), the bounded perturbation regularization (BPR) algorithm, and the generalized cross-validation (GCV) method. These methods are used to tune the regularization parameters based on linear estimation models, with the sample covariance matrix's square root being the linear operator. Results obtained from both synthetic and real data demonstrate the consistency and effectiveness of the proposed R2LDA approach, especially in scenarios involving test data contaminated with noise that is not observed during the training phase.

Related papers

• Synergistic eigenanalysis of covariance and Hessian matrices for enhanced binary classification [72.77513633290056]
  We present a novel approach that combines the eigenanalysis of a covariance matrix evaluated on a training set with a Hessian matrix evaluated on a deep learning model. Our method captures intricate patterns and relationships, enhancing classification performance.
  arXiv  Detail & Related papers  (2024-02-14T16:10:42Z)
• Regularized Linear Discriminant Analysis Using a Nonlinear Covariance Matrix Estimator [11.887333567383239]
  Linear discriminant analysis (LDA) is a widely used technique for data classification. LDA becomes inefficient when the data covariance matrix is ill-conditioned. Regularized LDA methods have been proposed to cope with such a situation.
  arXiv  Detail & Related papers  (2024-01-31T11:37:14Z)
• Minimally Informed Linear Discriminant Analysis: training an LDA model with unlabelled data [51.673443581397954]
  We show that it is possible to compute the exact projection vector from LDA models based on unlabelled data. We show that the MILDA projection vector can be computed in a closed form with a computational cost comparable to LDA.
  arXiv  Detail & Related papers  (2023-10-17T09:50:31Z)
• Offline Policy Optimization in RL with Variance Regularizaton [142.87345258222942]
  We propose variance regularization for offline RL algorithms, using stationary distribution corrections. We show that by using Fenchel duality, we can avoid double sampling issues for computing the gradient of the variance regularizer. The proposed algorithm for offline variance regularization (OVAR) can be used to augment any existing offline policy optimization algorithms.
  arXiv  Detail & Related papers  (2022-12-29T18:25:01Z)
• Varying Coefficient Linear Discriminant Analysis for Dynamic Data [5.228711636020666]
  This paper investigates the varying coefficient LDA model for dynamic data. By deriving a new discriminant direction function parallel with Bayes' direction, we propose a least-square estimation procedure. For high-dimensional regime, the corresponding data-driven discriminant rule is more computationally efficient than the existed dynamic linear programming rule.
  arXiv  Detail & Related papers  (2022-03-12T07:32:19Z)
• Data adaptive RKHS Tikhonov regularization for learning kernels in operators [1.5039745292757671]
  We present DARTR: a Data Adaptive RKHS Tikhonov Regularization method for the linear inverse problem of nonparametric learning of function parameters in operators. A key ingredient is a system intrinsic data-adaptive (SIDA) RKHS, whose norm restricts the learning to take place in the function space of identifiability.
  arXiv  Detail & Related papers  (2022-03-08T01:08:35Z)
• A Priori Denoising Strategies for Sparse Identification of Nonlinear Dynamical Systems: A Comparative Study [68.8204255655161]
  We investigate and compare the performance of several local and global smoothing techniques to a priori denoise the state measurements. We show that, in general, global methods, which use the entire measurement data set, outperform local methods, which employ a neighboring data subset around a local point.
  arXiv  Detail & Related papers  (2022-01-29T23:31:25Z)
• Weight Vector Tuning and Asymptotic Analysis of Binary Linear Classifiers [82.5915112474988]
  This paper proposes weight vector tuning of a generic binary linear classifier through the parameterization of a decomposition of the discriminant by a scalar. It is also found that weight vector tuning significantly improves the performance of Linear Discriminant Analysis (LDA) under high estimation noise.
  arXiv  Detail & Related papers  (2021-10-01T17:50:46Z)
• Doubly Robust Semiparametric Difference-in-Differences Estimators with High-Dimensional Data [15.27393561231633]
  We propose a doubly robust two-stage semiparametric difference-in-difference estimator for estimating heterogeneous treatment effects. The first stage allows a general set of machine learning methods to be used to estimate the propensity score. In the second stage, we derive the rates of convergence for both the parametric parameter and the unknown function.
  arXiv  Detail & Related papers  (2020-09-07T15:14:29Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• Improving Covariance-Regularized Discriminant Analysis for EHR-based Predictive Analytics of Diseases [20.697847129363463]
  We study an analytical model that understands the accuracy of LDA for classifying data with arbitrary distribution. We also propose a novel LDA classifier De-Sparse that outperforms state-of-the-art LDA approaches developed for HDLSS data.
  arXiv  Detail & Related papers  (2016-10-18T06:11:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.