Robust Regression via Model Based Methods
- URL: http://arxiv.org/abs/2106.10759v2
- Date: Tue, 22 Jun 2021 02:24:13 GMT
- Title: Robust Regression via Model Based Methods
- Authors: Armin Moharrer, Khashayar Kamran, Edmund Yeh, and Stratis Ioannidis
- Abstract summary: We propose an algorithm inspired by so-called model-based optimization (MBO) [35, 36], which replaces a non-objective with a convex model function.
We apply this to robust regression, proposing SADM, a function of the Online Alternating Direction Method of Multipliers (OOADM) [50] to solve the inner optimization in MBO.
Finally, we demonstrate experimentally (a) the robustness of l_p norms to outliers and (b) the efficiency of our proposed model-based algorithms in comparison with methods on autoencoders and multi-target regression.
- Score: 13.300549123177705
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The mean squared error loss is widely used in many applications, including
auto-encoders, multi-target regression, and matrix factorization, to name a
few. Despite computational advantages due to its differentiability, it is not
robust to outliers. In contrast, l_p norms are known to be robust, but cannot
be optimized via, e.g., stochastic gradient descent, as they are
non-differentiable. We propose an algorithm inspired by so-called model-based
optimization (MBO) [35, 36], which replaces a non-convex objective with a
convex model function and alternates between optimizing the model function and
updating the solution. We apply this to robust regression, proposing SADM, a
stochastic variant of the Online Alternating Direction Method of Multipliers
(OADM) [50] to solve the inner optimization in MBO. We show that SADM converges
with the rate O(log T/T). Finally, we demonstrate experimentally (a) the
robustness of l_p norms to outliers and (b) the efficiency of our proposed
model-based algorithms in comparison with gradient methods on autoencoders and
multi-target regression.
Related papers
- Covariance-Adaptive Sequential Black-box Optimization for Diffusion Targeted Generation [60.41803046775034]
We show how to perform user-preferred targeted generation via diffusion models with only black-box target scores of users.
Experiments on both numerical test problems and target-guided 3D-molecule generation tasks show the superior performance of our method in achieving better target scores.
arXiv Detail & Related papers (2024-06-02T17:26:27Z) - Solving Inverse Problems with Model Mismatch using Untrained Neural Networks within Model-based Architectures [14.551812310439004]
We introduce an untrained forward model residual block within the model-based architecture to match the data consistency in the measurement domain for each instance.
Our approach offers a unified solution that is less parameter-sensitive, requires no additional data, and enables simultaneous fitting of the forward model and reconstruction in a single pass.
arXiv Detail & Related papers (2024-03-07T19:02:13Z) - Multi-Response Heteroscedastic Gaussian Process Models and Their
Inference [1.52292571922932]
We propose a novel framework for the modeling of heteroscedastic covariance functions.
We employ variational inference to approximate the posterior and facilitate posterior predictive modeling.
We show that our proposed framework offers a robust and versatile tool for a wide array of applications.
arXiv Detail & Related papers (2023-08-29T15:06:47Z) - Variational Laplace Autoencoders [53.08170674326728]
Variational autoencoders employ an amortized inference model to approximate the posterior of latent variables.
We present a novel approach that addresses the limited posterior expressiveness of fully-factorized Gaussian assumption.
We also present a general framework named Variational Laplace Autoencoders (VLAEs) for training deep generative models.
arXiv Detail & Related papers (2022-11-30T18:59:27Z) - Inverse Kernel Decomposition [3.066967635405937]
We propose a novel nonlinear dimensionality reduction method -- Inverse Kernel Decomposition (IKD)
IKD is based on an eigen-decomposition of the sample covariance matrix of data.
We use synthetic datasets and four real-world datasets to show that IKD is a better dimensionality reduction method than other eigen-decomposition-based methods.
arXiv Detail & Related papers (2022-11-11T02:14:29Z) - Sparse high-dimensional linear regression with a partitioned empirical
Bayes ECM algorithm [62.997667081978825]
We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression.
Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates.
The proposed approach is implemented in the R package probe.
arXiv Detail & Related papers (2022-09-16T19:15:50Z) - MARS via LASSO [1.5199437137239338]
We propose and study a natural variant of the MARS method.
Our method is based on at least squares estimation over a convex class of functions.
Our estimator can be computed via finite-dimensional convex optimization.
arXiv Detail & Related papers (2021-11-23T07:30:33Z) - On Stochastic Moving-Average Estimators for Non-Convex Optimization [105.22760323075008]
In this paper, we demonstrate the power of a widely used estimator based on moving average (SEMA) problems.
For all these-the-art results, we also present the results for all these-the-art problems.
arXiv Detail & Related papers (2021-04-30T08:50:24Z) - Robust, Accurate Stochastic Optimization for Variational Inference [68.83746081733464]
We show that common optimization methods lead to poor variational approximations if the problem is moderately large.
Motivated by these findings, we develop a more robust and accurate optimization framework by viewing the underlying algorithm as producing a Markov chain.
arXiv Detail & Related papers (2020-09-01T19:12:11Z) - Efficient Learning of Generative Models via Finite-Difference Score
Matching [111.55998083406134]
We present a generic strategy to efficiently approximate any-order directional derivative with finite difference.
Our approximation only involves function evaluations, which can be executed in parallel, and no gradient computations.
arXiv Detail & Related papers (2020-07-07T10:05:01Z) - Uncertainty Modelling in Risk-averse Supply Chain Systems Using
Multi-objective Pareto Optimization [0.0]
One of the arduous tasks in supply chain modelling is to build robust models against irregular variations.
We have introduced a novel methodology namely, Pareto Optimization to handle uncertainties and bound the entropy of such uncertainties by explicitly modelling them under some apriori assumptions.
arXiv Detail & Related papers (2020-04-24T21:04:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.