Supervised Dictionary Learning with Auxiliary Covariates
- URL: http://arxiv.org/abs/2206.06774v1
- Date: Tue, 14 Jun 2022 12:10:03 GMT
- Title: Supervised Dictionary Learning with Auxiliary Covariates
- Authors: Joowon Lee, Hanbaek Lyu, Weixin Yao
- Abstract summary: Supervised dictionary learning (SDL) is a machine learning method that simultaneously seeks feature extraction and classification tasks.
We provide a novel framework that lifts' SDL as a convex problem in a combined factor space.
We apply SDL for imbalanced document classification by supervised topic modeling and also for pneumonia from chest X-ray images.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Supervised dictionary learning (SDL) is a classical machine learning method
that simultaneously seeks feature extraction and classification tasks, which
are not necessarily a priori aligned objectives. The goal of SDL is to learn a
class-discriminative dictionary, which is a set of latent feature vectors that
can well-explain both the features as well as labels of observed data. In this
paper, we provide a systematic study of SDL, including the theory, algorithm,
and applications of SDL. First, we provide a novel framework that `lifts' SDL
as a convex problem in a combined factor space and propose a low-rank projected
gradient descent algorithm that converges exponentially to the global minimizer
of the objective. We also formulate generative models of SDL and provide global
estimation guarantees of the true parameters depending on the hyperparameter
regime. Second, viewed as a nonconvex constrained optimization problem, we
provided an efficient block coordinate descent algorithm for SDL that is
guaranteed to find an $\varepsilon$-stationary point of the objective in
$O(\varepsilon^{-1}(\log \varepsilon^{-1})^{2})$ iterations. For the
corresponding generative model, we establish a novel non-asymptotic local
consistency result for constrained and regularized maximum likelihood
estimation problems, which may be of independent interest. Third, we apply SDL
for imbalanced document classification by supervised topic modeling and also
for pneumonia detection from chest X-ray images. We also provide simulation
studies to demonstrate that SDL becomes more effective when there is a
discrepancy between the best reconstructive and the best discriminative
dictionaries.
Related papers
- Fusing Dictionary Learning and Support Vector Machines for Unsupervised Anomaly Detection [1.5999407512883508]
We introduce a new anomaly detection model that unifies the OC-SVM and DL residual functions into a single composite objective.
We extend both objectives to the more general setting that allows the use of kernel functions.
arXiv Detail & Related papers (2024-04-05T12:41:53Z) - Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
We propose an effective method called Latent Semantic Consensus (LSC)
LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses.
LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
arXiv Detail & Related papers (2024-03-11T05:35:38Z) - Synergistic eigenanalysis of covariance and Hessian matrices for enhanced binary classification [72.77513633290056]
We present a novel approach that combines the eigenanalysis of a covariance matrix evaluated on a training set with a Hessian matrix evaluated on a deep learning model.
Our method captures intricate patterns and relationships, enhancing classification performance.
arXiv Detail & Related papers (2024-02-14T16:10:42Z) - Random Matrix Analysis to Balance between Supervised and Unsupervised
Learning under the Low Density Separation Assumption [9.620832983703863]
We introduce QLDS, a linear classification model, where the low density separation assumption is implemented via quadratic margin.
We show that particular cases of our algorithm are the least-square support vector machine in the supervised case, the spectral clustering in the fully unsupervised regime, and a class of semi-supervised graph-based approaches.
arXiv Detail & Related papers (2023-10-20T11:46:12Z) - Stochastic Unrolled Federated Learning [85.6993263983062]
We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning.
Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
arXiv Detail & Related papers (2023-05-24T17:26:22Z) - Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via
GDPA Linearization [59.87663954467815]
Algorithm unfolding creates an interpretable and parsimonious neural network architecture by implementing each iteration of a model-based algorithm as a neural layer.
In this paper, leveraging a recent linear algebraic theorem called Gershgorin disc perfect alignment (GDPA), we unroll a projection-free algorithm for semi-definite programming relaxation (SDR) of a binary graph.
Experimental results show that our unrolled network outperformed pure model-based graph classifiers, and achieved comparable performance to pure data-driven networks but using far fewer parameters.
arXiv Detail & Related papers (2021-09-10T07:01:15Z) - UCSL : A Machine Learning Expectation-Maximization framework for
Unsupervised Clustering driven by Supervised Learning [2.133032470368051]
Subtype Discovery consists in finding interpretable and consistent sub-parts of a dataset, which are also relevant to a certain supervised task.
We propose a general Expectation-Maximization ensemble framework entitled UCSL (Unsupervised Clustering driven by Supervised Learning)
Our method is generic, it can integrate any clustering method and can be driven by both binary classification and regression.
arXiv Detail & Related papers (2021-07-05T12:55:13Z) - Generalized One-Class Learning Using Pairs of Complementary Classifiers [41.64645294104883]
One-class learning is the classic problem of fitting a model to the data for which annotations are available only for a single class.
In this paper, we explore novel objectives for one-class learning, which we collectively refer to as Generalized One-class Discriminative Subspaces (GODS)
arXiv Detail & Related papers (2021-06-24T18:52:05Z) - Learning while Respecting Privacy and Robustness to Distributional
Uncertainties and Adversarial Data [66.78671826743884]
The distributionally robust optimization framework is considered for training a parametric model.
The objective is to endow the trained model with robustness against adversarially manipulated input data.
Proposed algorithms offer robustness with little overhead.
arXiv Detail & Related papers (2020-07-07T18:25:25Z) - Progressive Identification of True Labels for Partial-Label Learning [112.94467491335611]
Partial-label learning (PLL) is a typical weakly supervised learning problem, where each training instance is equipped with a set of candidate labels among which only one is the true label.
Most existing methods elaborately designed as constrained optimizations that must be solved in specific manners, making their computational complexity a bottleneck for scaling up to big data.
This paper proposes a novel framework of classifier with flexibility on the model and optimization algorithm.
arXiv Detail & Related papers (2020-02-19T08:35:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.