A Bi-level Nonlinear Eigenvector Algorithm for Wasserstein Discriminant Analysis

• URL: http://arxiv.org/abs/2211.11891v2
• Date: Fri, 28 Jul 2023 02:07:41 GMT
• Title: A Bi-level Nonlinear Eigenvector Algorithm for Wasserstein Discriminant Analysis
• Authors: Dong Min Roh, Zhaojun Bai, Ren-Cang Li
• Abstract summary: Wasserstein discriminant analysis (WDA) is a linear dimensionality reduction method. WDA can account for both global and local interconnections between data classes. A bi-level nonlinear eigenvector algorithm (WDA-nepv) is presented.
• Score: 3.4806267677524896
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Much like the classical Fisher linear discriminant analysis (LDA), the recently proposed Wasserstein discriminant analysis (WDA) is a linear dimensionality reduction method that seeks a projection matrix to maximize the dispersion of different data classes and minimize the dispersion of same data classes via a bi-level optimization. In contrast to LDA, WDA can account for both global and local interconnections between data classes by using the underlying principles of optimal transport. In this paper, a bi-level nonlinear eigenvector algorithm (WDA-nepv) is presented to fully exploit the structures of the bi-level optimization of WDA. The inner level of WDA-nepv for computing the optimal transport matrices is formulated as an eigenvector-dependent nonlinear eigenvalue problem (NEPv), and meanwhile, the outer level for trace ratio optimizations is formulated as another NEPv. Both NEPvs can be computed efficiently under the self-consistent field (SCF) framework. WDA-nepv is derivative-free and surrogate-model-free when compared with existing algorithms. Convergence analysis of the proposed WDA-nepv justifies the utilization of the SCF for solving the bi-level optimization of WDA. Numerical experiments with synthetic and real-life datasets demonstrate the classification accuracy and scalability of WDA-nepv.

Related papers

• Synergistic eigenanalysis of covariance and Hessian matrices for enhanced binary classification [72.77513633290056]
  We present a novel approach that combines the eigenanalysis of a covariance matrix evaluated on a training set with a Hessian matrix evaluated on a deep learning model. Our method captures intricate patterns and relationships, enhancing classification performance.
  arXiv  Detail & Related papers  (2024-02-14T16:10:42Z)
• Regularized Linear Discriminant Analysis Using a Nonlinear Covariance Matrix Estimator [11.887333567383239]
  Linear discriminant analysis (LDA) is a widely used technique for data classification. LDA becomes inefficient when the data covariance matrix is ill-conditioned. Regularized LDA methods have been proposed to cope with such a situation.
  arXiv  Detail & Related papers  (2024-01-31T11:37:14Z)
• A Preconditioned Interior Point Method for Support Vector Machines Using an ANOVA-Decomposition and NFFT-Based Matrix-Vector Products [0.6445605125467574]
  We propose an NFFT-accelerated matrix-vector product using an ANOVA decomposition for the feature space that is used within an interior point method for the overall optimization problem. We investigate the performance of the different preconditioners as well as the accuracy of the ANOVA kernel on several large-scale datasets.
  arXiv  Detail & Related papers  (2023-12-01T12:27:11Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Minimally Informed Linear Discriminant Analysis: training an LDA model with unlabelled data [51.673443581397954]
  We show that it is possible to compute the exact projection vector from LDA models based on unlabelled data. We show that the MILDA projection vector can be computed in a closed form with a computational cost comparable to LDA.
  arXiv  Detail & Related papers  (2023-10-17T09:50:31Z)
• GO-LDA: Generalised Optimal Linear Discriminant Analysis [6.644357197885522]
  Linear discriminant analysis has been a useful tool in pattern recognition and data analysis research and practice. We show that the generalised eigenanalysis solution to multiclass LDA does neither yield orthogonal discriminant directions nor maximise discrimination of projected data along them.
  arXiv  Detail & Related papers  (2023-05-23T23:11:05Z)
• Spectrally-Corrected and Regularized Linear Discriminant Analysis for Spiked Covariance Model [2.517838307493912]
  This paper proposes an improved linear discriminant analysis called spectrally-corrected and regularized LDA (SRLDA) It is proved that SRLDA has a linear classification global optimal solution under the spiked model assumption. Experiments on different data sets show that the SRLDA algorithm performs better in classification and dimensionality reduction than currently used tools.
  arXiv  Detail & Related papers  (2022-10-08T00:47:50Z)
• coVariance Neural Networks [119.45320143101381]
  Graph neural networks (GNN) are an effective framework that exploit inter-relationships within graph-structured data for learning. We propose a GNN architecture, called coVariance neural network (VNN), that operates on sample covariance matrices as graphs. We show that VNN performance is indeed more stable than PCA-based statistical approaches.
  arXiv  Detail & Related papers  (2022-05-31T15:04:43Z)
• Varying Coefficient Linear Discriminant Analysis for Dynamic Data [5.228711636020666]
  This paper investigates the varying coefficient LDA model for dynamic data. By deriving a new discriminant direction function parallel with Bayes' direction, we propose a least-square estimation procedure. For high-dimensional regime, the corresponding data-driven discriminant rule is more computationally efficient than the existed dynamic linear programming rule.
  arXiv  Detail & Related papers  (2022-03-12T07:32:19Z)
• Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via GDPA Linearization [59.87663954467815]
  Algorithm unfolding creates an interpretable and parsimonious neural network architecture by implementing each iteration of a model-based algorithm as a neural layer. In this paper, leveraging a recent linear algebraic theorem called Gershgorin disc perfect alignment (GDPA), we unroll a projection-free algorithm for semi-definite programming relaxation (SDR) of a binary graph. Experimental results show that our unrolled network outperformed pure model-based graph classifiers, and achieved comparable performance to pure data-driven networks but using far fewer parameters.
  arXiv  Detail & Related papers  (2021-09-10T07:01:15Z)
• Bilevel Optimization: Convergence Analysis and Enhanced Design [63.64636047748605]
  Bilevel optimization is a tool for many machine learning problems. We propose a novel stoc-efficientgradient estimator named stoc-BiO.
  arXiv  Detail & Related papers  (2020-10-15T18:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.