Robustified Multivariate Regression and Classification Using
Distributionally Robust Optimization under the Wasserstein Metric
- URL: http://arxiv.org/abs/2006.06090v1
- Date: Wed, 10 Jun 2020 22:16:50 GMT
- Title: Robustified Multivariate Regression and Classification Using
Distributionally Robust Optimization under the Wasserstein Metric
- Authors: Ruidi Chen and Ioannis Ch. Paschalidis
- Abstract summary: We develop Distributionally Robust Optimization (DRO) formulations for Multivariate Linear Regression (MLR) and Multiclass Logistic Regression (MLG)
We relax the DRO formulation into a regularized learning problem whose regularizer is a norm of the coefficient matrix.
Experimental results show that our approach improves the predictive error by 7% -- 37% for MLR, and a metric of robustness by 100% for MLG.
- Score: 11.383869751239166
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We develop Distributionally Robust Optimization (DRO) formulations for
Multivariate Linear Regression (MLR) and Multiclass Logistic Regression (MLG)
when both the covariates and responses/labels may be contaminated by outliers.
The DRO framework uses a probabilistic ambiguity set defined as a ball of
distributions that are close to the empirical distribution of the training set
in the sense of the Wasserstein metric. We relax the DRO formulation into a
regularized learning problem whose regularizer is a norm of the coefficient
matrix. We establish out-of-sample performance guarantees for the solutions to
our model, offering insights on the role of the regularizer in controlling the
prediction error. Experimental results show that our approach improves the
predictive error by 7% -- 37% for MLR, and a metric of robustness by 100% for
MLG.
Related papers
- Metric-aware LLM inference for regression and scoring [52.764328080398805]
Large language models (LLMs) have demonstrated strong results on a range of NLP tasks.
We show that this inference strategy can be suboptimal for a range of regression and scoring tasks, and associated evaluation metrics.
We propose aware metric LLM inference: a decision theoretic approach optimizing for custom regression and scoring metrics at inference time.
arXiv Detail & Related papers (2024-03-07T03:24:34Z) - Modeling the Q-Diversity in a Min-max Play Game for Robust Optimization [61.39201891894024]
Group distributionally robust optimization (group DRO) can minimize the worst-case loss over pre-defined groups.
We reformulate the group DRO framework by proposing Q-Diversity.
Characterized by an interactive training mode, Q-Diversity relaxes the group identification from annotation into direct parameterization.
arXiv Detail & Related papers (2023-05-20T07:02:27Z) - Distributionally Robust Multiclass Classification and Applications in
Deep Image Classifiers [9.979945269265627]
We develop a Distributionally Robust Optimization (DRO) formulation for Multiclass Logistic Regression (MLR)
We demonstrate reductions in test error rate by up to 83.5% and loss by up to 91.3% compared with baseline methods, by adopting a novel random training method.
arXiv Detail & Related papers (2022-10-15T05:09:28Z) - Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios.
We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
arXiv Detail & Related papers (2022-04-13T12:43:12Z) - Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma
Distributions [91.63716984911278]
We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result.
Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
arXiv Detail & Related papers (2021-11-11T14:28:12Z) - Distributionally Robust Multi-Output Regression Ranking [3.9318191265352196]
We introduce a new listwise listwise learning-to-rank model called Distributionally Robust Multi-output Regression Ranking (DRMRR)
DRMRR uses a Distributionally Robust Optimization framework to minimize a multi-output loss function under the most adverse distributions in the neighborhood of the empirical data distribution.
Our experiments were conducted on two real-world applications, medical document retrieval, and drug response prediction.
arXiv Detail & Related papers (2021-09-27T05:19:27Z) - Distributionally Robust Multiclass Classification and Applications in
Deep Image Classifiers [3.179831861897336]
We develop a Distributionally Robust Optimization (DRO) formulation for Multiclass Logistic Regression (MLR)
We demonstrate reductions in test error rate by up to 83.5% and loss by up to 91.3% compared with baseline methods, by adopting a novel random training method.
arXiv Detail & Related papers (2021-09-27T02:58:19Z) - Distributionally Robust Learning [11.916893752969429]
This book develops a comprehensive statistical learning framework that is robust to (distributional) perturbations in the data.
A tractable DRO relaxation for each problem is being derived, establishing a connection between bounds and regularization.
Beyond theory, we include numerical experiments and case studies using synthetic and real data.
arXiv Detail & Related papers (2021-08-20T04:14:18Z) - DORO: Distributional and Outlier Robust Optimization [98.44757325531631]
We propose the framework of DORO, for Distributional and Outlier Robust Optimization.
At the core of this approach is a refined risk function which prevents DRO from overfitting to potential outliers.
We theoretically prove the effectiveness of the proposed method, and empirically show that DORO improves the performance and stability of DRO with experiments on large modern datasets.
arXiv Detail & Related papers (2021-06-11T02:59:54Z) - Residuals-based distributionally robust optimization with covariate
information [0.0]
We consider data-driven approaches that integrate a machine learning prediction model within distributionally robust optimization (DRO)
Our framework is flexible in the sense that it can accommodate a variety of learning setups and DRO ambiguity sets.
arXiv Detail & Related papers (2020-12-02T11:21:34Z) - Distributional Robustness and Regularization in Reinforcement Learning [62.23012916708608]
We introduce a new regularizer for empirical value functions and show that it lower bounds the Wasserstein distributionally robust value function.
It suggests using regularization as a practical tool for dealing with $textitexternal uncertainty$ in reinforcement learning.
arXiv Detail & Related papers (2020-03-05T19:56:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.