Mixture of Experts with Uncertainty Voting for Imbalanced Deep
Regression Problems
- URL: http://arxiv.org/abs/2305.15178v2
- Date: Thu, 30 Nov 2023 18:29:22 GMT
- Title: Mixture of Experts with Uncertainty Voting for Imbalanced Deep
Regression Problems
- Authors: Yuchang Jiang, Vivien Sainte Fare Garnot, Konrad Schindler, Jan Dirk
Wegner
- Abstract summary: We propose a mixture-of-experts approach to imbalanced regression problems.
We replace traditional regression losses with negative log-likelihood which also predicts sample-wise aleatoric uncertainty.
We show experimentally that such a loss handles the imbalance better.
- Score: 22.041067758144077
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Data imbalance is ubiquitous when applying machine learning to real-world
problems, particularly regression problems. If training data are imbalanced,
the learning is dominated by the densely covered regions of the target
distribution, consequently, the learned regressor tends to exhibit poor
performance in sparsely covered regions. Beyond standard measures like
over-sampling or re-weighting, there are two main directions to handle learning
from imbalanced data. For regression, recent work relies on the continuity of
the distribution; whereas for classification there has been a trend to employ
mixture-of-expert models and let some ensemble members specialize in
predictions for the sparser regions. In our method, dubbed MOUV, we propose to
leverage recent work on probabilistic deep learning and integrate it in a
mixture-of-experts approach for imbalanced regression. We replace traditional
regression losses with negative log-likelihood which also predicts sample-wise
aleatoric uncertainty. We show experimentally that such a loss handles the
imbalance better. Secondly, we use the readily available aleatoric uncertainty
values to fuse the predictions of a mixture-of-experts model, thus obviating
the need for a separate aggregation module. We compare our method with existing
alternatives on multiple public benchmarks and show that MOUV consistently
outperforms the prior art, while at the same time producing better calibrated
uncertainty estimates. Our code is available at link-upon-publication.
Related papers
- Beyond the Norms: Detecting Prediction Errors in Regression Models [26.178065248948773]
This paper tackles the challenge of detecting unreliable behavior in regression algorithms.
We introduce the notion of unreliability in regression, when the output of the regressor exceeds a specified discrepancy (or error)
We show empirical improvements in error detection for multiple regression tasks, consistently outperforming popular baseline approaches.
arXiv Detail & Related papers (2024-06-11T05:51:44Z) - Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise [51.87307904567702]
Quantile regression is a leading approach for obtaining such intervals via the empirical estimation of quantiles in the distribution of outputs.
We propose Relaxed Quantile Regression (RQR), a direct alternative to quantile regression based interval construction that removes this arbitrary constraint.
We demonstrate that this added flexibility results in intervals with an improvement in desirable qualities.
arXiv Detail & Related papers (2024-06-05T13:36:38Z) - Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data.
Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
arXiv Detail & Related papers (2024-04-24T09:04:36Z) - Imbalance in Regression Datasets [0.9374652839580183]
We argue that imbalance in regression is an equally important problem which has so far been overlooked.
Due to under- and over-representations in a data set's target distribution, regressors are prone to degenerate to naive models.
arXiv Detail & Related papers (2024-02-19T09:06:26Z) - Variational Imbalanced Regression: Fair Uncertainty Quantification via
Probabilistic Smoothing [13.339286071690394]
Existing regression models tend to fall short in both accuracy and uncertainty estimation when the label distribution is imbalanced.
We propose a probabilistic deep learning model, dubbed variational imbalanced regression (VIR)
VIR performs well in imbalanced regression but naturally produces reasonable uncertainty estimation as a byproduct.
arXiv Detail & Related papers (2023-06-11T06:27:06Z) - Learning Sample Difficulty from Pre-trained Models for Reliable
Prediction [55.77136037458667]
We propose to utilize large-scale pre-trained models to guide downstream model training with sample difficulty-aware entropy regularization.
We simultaneously improve accuracy and uncertainty calibration across challenging benchmarks.
arXiv Detail & Related papers (2023-04-20T07:29:23Z) - Variation-Incentive Loss Re-weighting for Regression Analysis on Biased
Data [8.115323786541078]
We aim to improve the accuracy of the regression analysis by addressing the data skewness/bias during model training.
We propose a Variation-Incentive Loss re-weighting method (VILoss) to optimize the gradient descent-based model training for regression analysis.
arXiv Detail & Related papers (2021-09-14T10:22:21Z) - Bayesian analysis of the prevalence bias: learning and predicting from
imbalanced data [10.659348599372944]
This paper lays the theoretical and computational framework for training models, and for prediction, in the presence of prevalence bias.
It offers an alternative to principled training losses and complements test-time procedures based on selecting an operating point from summary curves.
It integrates seamlessly in the current paradigm of (deep) learning using backpropagation and naturally with Bayesian models.
arXiv Detail & Related papers (2021-07-31T14:36:33Z) - Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware
Regression [91.3373131262391]
Uncertainty is the only certainty there is.
Traditionally, the direct regression formulation is considered and the uncertainty is modeled by modifying the output space to a certain family of probabilistic distributions.
How to model the uncertainty within the present-day technologies for regression remains an open issue.
arXiv Detail & Related papers (2021-03-25T06:56:09Z) - Unlabelled Data Improves Bayesian Uncertainty Calibration under
Covariate Shift [100.52588638477862]
We develop an approximate Bayesian inference scheme based on posterior regularisation.
We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
arXiv Detail & Related papers (2020-06-26T13:50:19Z) - Evaluating Prediction-Time Batch Normalization for Robustness under
Covariate Shift [81.74795324629712]
We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift.
We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness.
The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
arXiv Detail & Related papers (2020-06-19T05:08:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.