Generalized Kernel Regularized Least Squares
- URL: http://arxiv.org/abs/2209.14355v4
- Date: Fri, 8 Sep 2023 18:26:01 GMT
- Title: Generalized Kernel Regularized Least Squares
- Authors: Qing Chang, Max Goplerud
- Abstract summary: Regularized Least Squares (KRLS) is a popular method for flexibly estimating models that may have complex relationships between variables.
Existing approaches are inflexible and do not allow KRLS to be combined with theoretically-motivated extensions such as random effects, unregularized fixed effects, or non-Gaussian outcomes.
Our paper addresses both concerns by introducing generalized KRLS (gKRLS)
We demonstrate that gKRLS can be fit on datasets with tens of thousands of observations in under one minute.
- Score: 2.8727524550655117
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Kernel Regularized Least Squares (KRLS) is a popular method for flexibly
estimating models that may have complex relationships between variables.
However, its usefulness to many researchers is limited for two reasons. First,
existing approaches are inflexible and do not allow KRLS to be combined with
theoretically-motivated extensions such as random effects, unregularized fixed
effects, or non-Gaussian outcomes. Second, estimation is extremely
computationally intensive for even modestly sized datasets. Our paper addresses
both concerns by introducing generalized KRLS (gKRLS). We note that KRLS can be
re-formulated as a hierarchical model thereby allowing easy inference and
modular model construction where KRLS can be used alongside random effects,
splines, and unregularized fixed effects. Computationally, we also implement
random sketching to dramatically accelerate estimation while incurring a
limited penalty in estimation quality. We demonstrate that gKRLS can be fit on
datasets with tens of thousands of observations in under one minute. Further,
state-of-the-art techniques that require fitting the model over a dozen times
(e.g. meta-learners) can be estimated quickly.
Related papers
- Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study [61.64685376882383]
Counterfactual learning to rank (CLTR) has attracted extensive attention in the IR community for its ability to leverage massive logged user interaction data to train ranking models.
This paper investigates the robustness of existing CLTR models in complex and diverse situations.
We find that the DLA models and IPS-DCM show better robustness under various simulation settings than IPS-PBM and PRS with offline propensity estimation.
arXiv Detail & Related papers (2024-04-04T10:54:38Z) - Optimal Multi-Distribution Learning [88.3008613028333]
Multi-distribution learning seeks to learn a shared model that minimizes the worst-case risk across $k$ distinct data distributions.
We propose a novel algorithm that yields an varepsilon-optimal randomized hypothesis with a sample complexity on the order of (d+k)/varepsilon2.
arXiv Detail & Related papers (2023-12-08T16:06:29Z) - Deep Ensembles Meets Quantile Regression: Uncertainty-aware Imputation
for Time Series [49.992908221544624]
Time series data often exhibit numerous missing values, which is the time series imputation task.
Previous deep learning methods have been shown to be effective for time series imputation.
We propose a non-generative time series imputation method that produces accurate imputations with inherent uncertainty.
arXiv Detail & Related papers (2023-12-03T05:52:30Z) - Accelerating Generalized Linear Models by Trading off Computation for
Uncertainty [29.877181350448193]
Generalized Linear Models (GLMs) define a flexible probabilistic framework to model categorical, ordinal and continuous data.
The resulting approximation error adversely impacts the reliability of the model and is not accounted for in the uncertainty of the prediction.
We introduce a family of iterative methods that explicitly model this error.
arXiv Detail & Related papers (2023-10-31T08:58:16Z) - Posterior Coreset Construction with Kernelized Stein Discrepancy for
Model-Based Reinforcement Learning [78.30395044401321]
We develop a novel model-based approach to reinforcement learning (MBRL)
It relaxes the assumptions on the target transition model to belong to a generic family of mixture models.
It can achieve up-to 50 percent reduction in wall clock time in some continuous control environments.
arXiv Detail & Related papers (2022-06-02T17:27:49Z) - Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios.
We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
arXiv Detail & Related papers (2022-04-13T12:43:12Z) - Supervised Multivariate Learning with Simultaneous Feature Auto-grouping
and Dimension Reduction [7.093830786026851]
This paper proposes a novel clustered reduced-rank learning framework.
It imposes two joint matrix regularizations to automatically group the features in constructing predictive factors.
It is more interpretable than low-rank modeling and relaxes the stringent sparsity assumption in variable selection.
arXiv Detail & Related papers (2021-12-17T20:11:20Z) - Scaling Structured Inference with Randomization [64.18063627155128]
We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states.
Our method is widely applicable to classical DP-based inference.
It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
arXiv Detail & Related papers (2021-12-07T11:26:41Z) - Scalable Spatiotemporally Varying Coefficient Modelling with Bayesian Kernelized Tensor Regression [17.158289775348063]
Kernelized tensor Regression (BKTR) can be considered a new and scalable approach to modeling processes with low-rank cotemporal structure.
We conduct extensive experiments on both synthetic and real-world data sets, and our results confirm the superior performance and efficiency of BKTR for model estimation and inference.
arXiv Detail & Related papers (2021-08-31T19:22:23Z) - Oversampling Divide-and-conquer for Response-skewed Kernel Ridge
Regression [20.00435452480056]
We develop a novel response-adaptive partition strategy to overcome the limitation of the divide-and-conquer method.
We show the proposed estimate has a smaller mean squared error (AMSE) than that of the classical dacKRR estimate under mild conditions.
arXiv Detail & Related papers (2021-07-13T04:01:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.