Discriminative classification with generative features: bridging Naive Bayes and logistic regression

• URL: http://arxiv.org/abs/2512.01097v1
• Date: Sun, 30 Nov 2025 21:34:24 GMT
• Title: Discriminative classification with generative features: bridging Naive Bayes and logistic regression
• Authors: Zachary Terner, Alexander Petersen, Yuedong Wang,
• Abstract summary: We introduce Smart Bayes, a new classification framework that bridges generative and discriminative modeling.<n>From the generative perspective, Smart Bayes relaxes the fixed unit weights of Naive Bayes by allowing data-driven coefficients on density-ratio features.<n>From a discriminative perspective, it constructs transformed inputs as marginal log-density ratios that explicitly quantify how much more likely each feature value is under one class.
• Score: 43.205900007057146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Smart Bayes, a new classification framework that bridges generative and discriminative modeling by integrating likelihood-ratio-based generative features into a logistic-regression-style discriminative classifier. From the generative perspective, Smart Bayes relaxes the fixed unit weights of Naive Bayes by allowing data-driven coefficients on density-ratio features. From a discriminative perspective, it constructs transformed inputs as marginal log-density ratios that explicitly quantify how much more likely each feature value is under one class than another, thereby providing predictors with stronger class separation than the raw covariates. To support this framework, we develop a spline-based estimator for univariate log-density ratios that is flexible, robust, and computationally efficient. Through extensive simulations and real-data studies, Smart Bayes often outperforms both logistic regression and Naive Bayes. Our results highlight the potential of hybrid approaches that exploit generative structure to enhance discriminative performance.

Related papers

• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Variational empirical Bayes variable selection in high-dimensional logistic regression [2.4032899110671955]
  We develop a novel and computationally efficient variational approximation thereof.<n>One such novelty is that we develop this approximation directly for the marginal distribution on the model space, rather than on the regression coefficients themselves.<n>We demonstrate the method's strong performance in simulations, and prove that our variational approximation inherits the strong selection consistency property satisfied by the posterior distribution that it is approximating.
  arXiv  Detail & Related papers  (2025-02-14T19:57:13Z)
• Optimal Projections for Classification with Naive Bayes [5.953513005270837]
  We study the problem of obtaining an alternative basis for the factorisation of class conditional densities.<n>We formulate the problem as a projection pursuit to find the optimal linear projection on which to perform classification.<n>We find that the proposed approach substantially outperforms other popular probabilistic discriminant analysis models.
  arXiv  Detail & Related papers  (2024-09-09T14:05:30Z)
• Variational Classification [51.2541371924591]
  We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders. Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency. We induce a chosen latent distribution, instead of the implicit assumption found in a standard softmax layer.
  arXiv  Detail & Related papers  (2023-05-17T17:47:19Z)
• Robust Outlier Rejection for 3D Registration with Variational Bayes [70.98659381852787]
  We develop a novel variational non-local network-based outlier rejection framework for robust alignment. We propose a voting-based inlier searching strategy to cluster the high-quality hypothetical inliers for transformation estimation.
  arXiv  Detail & Related papers  (2023-04-04T03:48:56Z)
• Classification via score-based generative modelling [0.0]
  We investigated the application of score-based learning in discriminative and generative classification settings. We performed experiments on simulated and real-world datasets, demonstrating its effectiveness in achieving and improving binary classification performance.
  arXiv  Detail & Related papers  (2022-07-22T13:59:43Z)
• A Semi-Supervised Adaptive Discriminative Discretization Method Improving Discrimination Power of Regularized Naive Bayes [0.48342038441006785]
  We propose a semi-supervised adaptive discriminative discretization framework for naive Bayes. It could better estimate the data distribution by utilizing both labeled data and unlabeled data through pseudo-labeling techniques. The proposed method also significantly reduces the information loss during discretization by utilizing an adaptive discriminative discretization scheme.
  arXiv  Detail & Related papers  (2021-11-22T04:36:40Z)
• $\beta$-Cores: Robust Large-Scale Bayesian Data Summarization in the Presence of Outliers [14.918826474979587]
  The quality of classic Bayesian inference depends critically on whether observations conform with the assumed data generating model. We propose a variational inference method that, in a principled way, can simultaneously scale to large datasets. We illustrate the applicability of our approach in diverse simulated and real datasets, and various statistical models.
  arXiv  Detail & Related papers  (2020-08-31T13:47:12Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)
• Naive Feature Selection: a Nearly Tight Convex Relaxation for Sparse Naive Bayes [51.55826927508311]
  We propose a sparse version of naive Bayes, which can be used for feature selection.<n>We prove that our convex relaxation bounds becomes tight as the marginal contribution of additional features decreases.<n>Both binary and multinomial sparse models are solvable in time almost linear in problem size.
  arXiv  Detail & Related papers  (2019-05-23T19:30:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.