Deep Linear Discriminant Analysis Revisited

• URL: http://arxiv.org/abs/2601.01619v1
• Date: Sun, 04 Jan 2026 17:59:11 GMT
• Title: Deep Linear Discriminant Analysis Revisited
• Authors: Maxat Tezekbayev, Rustem Takhanov, Arman Bolatov, Zhenisbek Assylbekov,
• Abstract summary: We show that for unconstrained Deep Linear Discriminant Analysis (LDA) classifiers, maximum-likelihood training admits pathological solutions.<n>We introduce the emphDiscriminative Negative Log-Likelihood (DNLL) loss, which augments the LDA log-likelihood with a simple penalty on the mixture density.
• Score: 3.569867801312133
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We show that for unconstrained Deep Linear Discriminant Analysis (LDA) classifiers, maximum-likelihood training admits pathological solutions in which class means drift together, covariances collapse, and the learned representation becomes almost non-discriminative. Conversely, cross-entropy training yields excellent accuracy but decouples the head from the underlying generative model, leading to highly inconsistent parameter estimates. To reconcile generative structure with discriminative performance, we introduce the \emph{Discriminative Negative Log-Likelihood} (DNLL) loss, which augments the LDA log-likelihood with a simple penalty on the mixture density. DNLL can be interpreted as standard LDA NLL plus a term that explicitly discourages regions where several classes are simultaneously likely. Deep LDA trained with DNLL produces clean, well-separated latent spaces, matches the test accuracy of softmax classifiers on synthetic data and standard image benchmarks, and yields substantially better calibrated predictive probabilities, restoring a coherent probabilistic interpretation to deep discriminant models.

Related papers

• Probabilistic Deep Discriminant Analysis for Wind Blade Segmentation [18.108693090007748]
  We introduce Deep Discriminant Analysis (DDA), which directly optimize the Fisher criterion utilizing deep networks.<n>We present two stable DDA loss functions and augment them with a probability loss, resulting in Probabilistic DDA (PDDA)<n>PDDA effectively minimizes class overlap in output distributions, producing highly confident predictions with reduced within-class variance.
  arXiv  Detail & Related papers  (2026-01-20T11:03:51Z)
• Simplex Deep Linear Discriminant Analysis [2.6381163133447836]
  We revisit Deep Linear Discriminant Analysis (Deep LDA) from a likelihood-based perspective.<n>While classical LDA is a simple Gaussian model with linear decision boundaries, attaching an LDA head to a neural encoder raises the question of how to train the resulting deep classifier by maximum likelihood estimation (MLE)<n>We first show that end-to-end MLE training of an unconstrained Deep LDA model ignores discrimination: when both the LDA parameters and the encoder parameters are learned jointly, the likelihood admits a degenerate solution in which some of the class clusters may heavily overlap or even collapse, and classification performance deteriorates
  arXiv  Detail & Related papers  (2026-01-04T22:22:59Z)
• Directly Handling Missing Data in Linear Discriminant Analysis for Enhancing Classification Accuracy and Interpretability [1.4840867281815378]
  We introduce a novel and robust classification method, termed weighted missing Linear Discriminant Analysis (WLDA) WLDA extends Linear Discriminant Analysis (LDA) to handle datasets with missing values without the need for imputation. We conduct an in-depth theoretical analysis to establish the properties of WLDA and thoroughly evaluate its explainability.
  arXiv  Detail & Related papers  (2024-06-30T14:21:32Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• GO-LDA: Generalised Optimal Linear Discriminant Analysis [6.644357197885522]
  Linear discriminant analysis has been a useful tool in pattern recognition and data analysis research and practice. We show that the generalised eigenanalysis solution to multiclass LDA does neither yield orthogonal discriminant directions nor maximise discrimination of projected data along them.
  arXiv  Detail & Related papers  (2023-05-23T23:11:05Z)
• Reusing the Task-specific Classifier as a Discriminator: Discriminator-free Adversarial Domain Adaptation [55.27563366506407]
  We introduce a discriminator-free adversarial learning network (DALN) for unsupervised domain adaptation (UDA) DALN achieves explicit domain alignment and category distinguishment through a unified objective. DALN compares favorably against the existing state-of-the-art (SOTA) methods on a variety of public datasets.
  arXiv  Detail & Related papers  (2022-04-08T04:40:18Z)
• Regularized Deep Linear Discriminant Analysis [26.08062442399418]
  As a non-linear extension of the classic Linear Discriminant Analysis(LDA), Deep Linear Discriminant Analysis(DLDA) replaces the original Categorical Cross Entropy(CCE) loss function. Regularization method on within-class scatter matrix is proposed to strengthen the discriminative ability of each dimension.
  arXiv  Detail & Related papers  (2021-05-15T03:54:32Z)
• Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures [100.55816326422773]
  We study the phenomenon of the maximum margin classifier for linear classification problems. Our results precisely characterize the condition under which benign overfitting can occur.
  arXiv  Detail & Related papers  (2021-04-28T08:25:16Z)
• General stochastic separation theorems with optimal bounds [68.8204255655161]
  Phenomenon of separability was revealed and used in machine learning to correct errors of Artificial Intelligence (AI) systems and analyze AI instabilities. Errors or clusters of errors can be separated from the rest of the data. The ability to correct an AI system also opens up the possibility of an attack on it, and the high dimensionality induces vulnerabilities caused by the same separability.
  arXiv  Detail & Related papers  (2020-10-11T13:12:41Z)
• Saliency-based Weighted Multi-label Linear Discriminant Analysis [101.12909759844946]
  We propose a new variant of Linear Discriminant Analysis (LDA) to solve multi-label classification tasks. The proposed method is based on a probabilistic model for defining the weights of individual samples. The Saliency-based weighted Multi-label LDA approach is shown to lead to performance improvements in various multi-label classification problems.
  arXiv  Detail & Related papers  (2020-04-08T19:40:53Z)
• When Relation Networks meet GANs: Relation GANs with Triplet Loss [110.7572918636599]
  Training stability is still a lingering concern of generative adversarial networks (GANs) In this paper, we explore a relation network architecture for the discriminator and design a triplet loss which performs better generalization and stability. Experiments on benchmark datasets show that the proposed relation discriminator and new loss can provide significant improvement on variable vision tasks.
  arXiv  Detail & Related papers  (2020-02-24T11:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.