Scalable Kernel Logistic Regression with Nystr\"om Approximation:
Theoretical Analysis and Application to Discrete Choice Modelling
- URL: http://arxiv.org/abs/2402.06763v1
- Date: Fri, 9 Feb 2024 19:52:31 GMT
- Title: Scalable Kernel Logistic Regression with Nystr\"om Approximation:
Theoretical Analysis and Application to Discrete Choice Modelling
- Authors: Jos\'e \'Angel Mart\'in-Baos, Ricardo Garc\'ia-R\'odenas, Luis
Rodriguez-Benitez, Michel Bierlaire
- Abstract summary: This paper introduces the Nystr"om approximation for Kernel Logistic Regression (KLR) on large datasets.
Four landmark selection methods are tested, including basic uniform sampling, a k-means sampling strategy, and two non-uniform methods grounded in leverage scores.
The performance of these strategies is evaluated using large-scale transport mode choice datasets.
- Score: 1.208453901299241
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The application of kernel-based Machine Learning (ML) techniques to discrete
choice modelling using large datasets often faces challenges due to memory
requirements and the considerable number of parameters involved in these
models. This complexity hampers the efficient training of large-scale models.
This paper addresses these problems of scalability by introducing the Nystr\"om
approximation for Kernel Logistic Regression (KLR) on large datasets. The study
begins by presenting a theoretical analysis in which: i) the set of KLR
solutions is characterised, ii) an upper bound to the solution of KLR with
Nystr\"om approximation is provided, and finally iii) a specialisation of the
optimisation algorithms to Nystr\"om KLR is described. After this, the
Nystr\"om KLR is computationally validated. Four landmark selection methods are
tested, including basic uniform sampling, a k-means sampling strategy, and two
non-uniform methods grounded in leverage scores. The performance of these
strategies is evaluated using large-scale transport mode choice datasets and is
compared with traditional methods such as Multinomial Logit (MNL) and
contemporary ML techniques. The study also assesses the efficiency of various
optimisation techniques for the proposed Nystr\"om KLR model. The performance
of gradient descent, Momentum, Adam, and L-BFGS-B optimisation methods is
examined on these datasets. Among these strategies, the k-means Nystr\"om KLR
approach emerges as a successful solution for applying KLR to large datasets,
particularly when combined with the L-BFGS-B and Adam optimisation methods. The
results highlight the ability of this strategy to handle datasets exceeding
200,000 observations while maintaining robust performance.
Related papers
- Minimally Supervised Learning using Topological Projections in
Self-Organizing Maps [55.31182147885694]
We introduce a semi-supervised learning approach based on topological projections in self-organizing maps (SOMs)
Our proposed method first trains SOMs on unlabeled data and then a minimal number of available labeled data points are assigned to key best matching units (BMU)
Our results indicate that the proposed minimally supervised model significantly outperforms traditional regression techniques.
arXiv Detail & Related papers (2024-01-12T22:51:48Z) - Large-scale Fully-Unsupervised Re-Identification [78.47108158030213]
We propose two strategies to learn from large-scale unlabeled data.
The first strategy performs a local neighborhood sampling to reduce the dataset size in each without violating neighborhood relationships.
A second strategy leverages a novel Re-Ranking technique, which has a lower time upper bound complexity and reduces the memory complexity from O(n2) to O(kn) with k n.
arXiv Detail & Related papers (2023-07-26T16:19:19Z) - Maximize to Explore: One Objective Function Fusing Estimation, Planning,
and Exploration [87.53543137162488]
We propose an easy-to-implement online reinforcement learning (online RL) framework called textttMEX.
textttMEX integrates estimation and planning components while balancing exploration exploitation automatically.
It can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards.
arXiv Detail & Related papers (2023-05-29T17:25:26Z) - Learning to Select Pivotal Samples for Meta Re-weighting [12.73177872962048]
We study how to learn to identify such a meta sample set from a large, imperfect training set, that is subsequently cleaned and used to optimize performance.
We propose two clustering methods within our learning framework, Representation-based clustering method (RBC) and Gradient-based clustering method (GBC)
arXiv Detail & Related papers (2023-02-09T03:04:40Z) - A distribution-free mixed-integer optimization approach to hierarchical modelling of clustered and longitudinal data [0.0]
We introduce an innovative algorithm that evaluates cluster effects for new data points, thereby increasing the robustness and precision of this model.
The inferential and predictive efficacy of this approach is further illustrated through its application in student scoring and protein expression.
arXiv Detail & Related papers (2023-02-06T23:34:51Z) - Regularization and Optimization in Model-Based Clustering [4.096453902709292]
k-means algorithm variants essentially fit a mixture of identical spherical Gaussians to data that vastly deviates from such a distribution.
We develop more effective optimization algorithms for general GMMs, and we combine these algorithms with regularization strategies that avoid overfitting.
These results shed new light on the current status quo between GMM and k-means methods and suggest the more frequent use of general GMMs for data exploration.
arXiv Detail & Related papers (2023-02-05T18:22:29Z) - Sparse high-dimensional linear regression with a partitioned empirical
Bayes ECM algorithm [62.997667081978825]
We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression.
Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates.
The proposed approach is implemented in the R package probe.
arXiv Detail & Related papers (2022-09-16T19:15:50Z) - Exploiting Temporal Structures of Cyclostationary Signals for
Data-Driven Single-Channel Source Separation [98.95383921866096]
We study the problem of single-channel source separation (SCSS)
We focus on cyclostationary signals, which are particularly suitable in a variety of application domains.
We propose a deep learning approach using a U-Net architecture, which is competitive with the minimum MSE estimator.
arXiv Detail & Related papers (2022-08-22T14:04:56Z) - Learning Distributionally Robust Models at Scale via Composite
Optimization [45.47760229170775]
We show how different variants of DRO are simply instances of a finite-sum composite optimization for which we provide scalable methods.
We also provide empirical results that demonstrate the effectiveness of our proposed algorithm with respect to the prior art in order to learn robust models from very large datasets.
arXiv Detail & Related papers (2022-03-17T20:47:42Z) - A Manifold Proximal Linear Method for Sparse Spectral Clustering with
Application to Single-Cell RNA Sequencing Data Analysis [9.643152256249884]
This paper considers the widely adopted SSC model as an optimization model with nonsmooth and non objective.
We propose a new method (ManPL) that solves the original SSC problem.
Results of the proposed methods are established.
arXiv Detail & Related papers (2020-07-18T22:05:00Z) - Clustering Binary Data by Application of Combinatorial Optimization
Heuristics [52.77024349608834]
We study clustering methods for binary data, first defining aggregation criteria that measure the compactness of clusters.
Five new and original methods are introduced, using neighborhoods and population behavior optimization metaheuristics.
From a set of 16 data tables generated by a quasi-Monte Carlo experiment, a comparison is performed for one of the aggregations using L1 dissimilarity, with hierarchical clustering, and a version of k-means: partitioning around medoids or PAM.
arXiv Detail & Related papers (2020-01-06T23:33:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.