Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression models

• URL: http://arxiv.org/abs/2401.10690v2
• Date: Fri, 5 Jul 2024 10:19:36 GMT
• Title: Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression models
• Authors: Jorge Paz-Ruza, Amparo Alonso-Betanzos, Bertha Guijarro-Berdiñas, Brais Cancela, Carlos Eiras-Franco,
• Abstract summary: We show that non-uniformity in the observed value distributions of individual entities leads to severely biased predictions in state-of-the-art models. We introduce Eccentricity-Area Under the Curve (EAUC) as a new metric that can quantify it in all studied models and datasets.
• Score: 5.336076422485076
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Dyadic regression models, which predict real-valued outcomes for pairs of entities, are fundamental in many domains (e.g. predicting the rating of a user to a product in Recommender Systems) and promising and under exploration in many others (e.g. approximating the adequate dosage of a drug for a patient in personalized pharmacology). In this work, we demonstrate that non-uniformity in the observed value distributions of individual entities leads to severely biased predictions in state-of-the-art models, skewing predictions towards the average of observed past values for the entity and providing worse-than-random predictive power in eccentric yet equally important cases. We show that the usage of global error metrics like Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) is insufficient to capture this phenomenon, which we name eccentricity bias, and we introduce Eccentricity-Area Under the Curve (EAUC) as a new complementary metric that can quantify it in all studied models and datasets. We also prove the adequateness of EAUC by using naive de-biasing corrections to demonstrate that a lower model bias correlates with a lower EAUC and vice-versa. This work contributes a bias-aware evaluation of dyadic regression models to avoid potential unfairness and risks in critical real-world applications of such systems.

Related papers

• MedDiffusion: Boosting Health Risk Prediction via Diffusion-based Data Augmentation [58.93221876843639]
  This paper introduces a novel, end-to-end diffusion-based risk prediction model, named MedDiffusion. It enhances risk prediction performance by creating synthetic patient data during training to enlarge sample space. It discerns hidden relationships between patient visits using a step-wise attention mechanism, enabling the model to automatically retain the most vital information for generating high-quality data.
  arXiv  Detail & Related papers  (2023-10-04T01:36:30Z)
• Biases in Inverse Ising Estimates of Near-Critical Behaviour [0.0]
  Inverse inference allows pairwise interactions to be reconstructed from empirical correlations. We show that estimators used for this inference, such as Pseudo-likelihood (PLM), are biased. Data-driven methods are explored and applied to a functional magnetic resonance imaging (fMRI) dataset from neuroscience.
  arXiv  Detail & Related papers  (2023-01-13T14:01:43Z)
• Bias-inducing geometries: an exactly solvable data model with fairness implications [13.690313475721094]
  We introduce an exactly solvable high-dimensional model of data imbalance. We analytically unpack the typical properties of learning models trained in this synthetic framework. We obtain exact predictions for the observables that are commonly employed for fairness assessment.
  arXiv  Detail & Related papers  (2022-05-31T16:27:57Z)
• Identifying and mitigating bias in algorithms used to manage patients in a pandemic [4.756860520861679]
  Logistic regression models were created to predict COVID-19 mortality, ventilator status and inpatient status using a real-world dataset. Models showed a 57% decrease in the number of biased trials. After calibration, the average sensitivity of the predictive models increased from 0.527 to 0.955.
  arXiv  Detail & Related papers  (2021-10-30T21:10:56Z)
• OMASGAN: Out-of-Distribution Minimum Anomaly Score GAN for Sample Generation on the Boundary [0.0]
  Generative models set high likelihood and low reconstruction loss to Out-of-Distribution (OoD) samples. OMASGAN generates, in a negative data augmentation manner, anomalous samples on the estimated distribution boundary. OMASGAN performs retraining by including the abnormal minimum-anomaly-score OoD samples generated on the distribution boundary.
  arXiv  Detail & Related papers  (2021-10-28T16:35:30Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• Loss Estimators Improve Model Generalization [36.520569284970456]
  We propose to train a loss estimator alongside the predictive model, using a contrastive training objective, to directly estimate the prediction uncertainties. We show the impact of loss estimators on model generalization, in terms of both its fidelity on in-distribution data and its ability to detect out of distribution samples or new classes unseen during training.
  arXiv  Detail & Related papers  (2021-03-05T16:35:10Z)
• UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data [81.00385374948125]
  We present UNcertaInTy-based hEalth risk prediction (UNITE) model. UNITE provides accurate disease risk prediction and uncertainty estimation leveraging multi-sourced health data. We evaluate UNITE on real-world disease risk prediction tasks: nonalcoholic fatty liver disease (NASH) and Alzheimer's disease (AD) UNITE achieves up to 0.841 in F1 score for AD detection, up to 0.609 in PR-AUC for NASH detection, and outperforms various state-of-the-art baselines by up to $19%$ over the best baseline.
  arXiv  Detail & Related papers  (2020-10-22T02:28:11Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.