Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression models

• URL: http://arxiv.org/abs/2401.10690v3
• Date: Mon, 30 Dec 2024 18:21:53 GMT
• Title: Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression models
• Authors: Jorge Paz-Ruza, Amparo Alonso-Betanzos, Bertha Guijarro-Berdiñas, Brais Cancela, Carlos Eiras-Franco,
• Abstract summary: Non-uniform observed value distributions of individual entities lead to severe biases in state-of-the-art models. We show that global error metrics like Root Mean Squared Error (RMSE) are insufficient to capture this bias. We introduce Eccentricity-Area Under the Curve (EAUC) as a novel complementary metric that can quantify it in all studied domains and models.
• Score: 5.336076422485076
• License:
• Abstract: Dyadic regression models, which output real-valued predictions for pairs of entities, are fundamental in many domains (e.g. obtaining user-product ratings in Recommender Systems) and promising and under exploration in others (e.g. tuning patient-drug dosages in personalized pharmacology). In this work, we prove that non-uniform observed value distributions of individual entities lead to severe biases in state-of-the-art models, skewing predictions towards the average of observed past values for the entity and providing worse-than-random predictive power in eccentric yet crucial cases; we name this phenomenon eccentricity bias. We show that global error metrics like Root Mean Squared Error (RMSE) are insufficient to capture this bias, and we introduce Eccentricity-Area Under the Curve (EAUC) as a novel complementary metric that can quantify it in all studied domains and models. We prove the intuitive interpretation of EAUC by experimenting with naive post-training bias corrections, and theorize other options to use EAUC to guide the construction of fair models. This work contributes a bias-aware evaluation of dyadic regression to prevent unfairness in critical real-world applications of such systems.

Related papers

• Editable Fairness: Fine-Grained Bias Mitigation in Language Models [52.66450426729818]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases. FAST surpasses state-of-the-art baselines with superior debiasing performance. This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• Fine-Grained Dynamic Framework for Bias-Variance Joint Optimization on Data Missing Not at Random [2.8165314121189247]
  In most practical applications such as recommendation systems, display advertising, and so forth, the collected data often contains missing values. We develop a systematic fine-grained dynamic learning framework to jointly optimize bias and variance.
  arXiv  Detail & Related papers  (2024-05-24T10:07:09Z)
• Learning for Counterfactual Fairness from Observational Data [62.43249746968616]
  Fairness-aware machine learning aims to eliminate biases of learning models against certain subgroups described by certain protected (sensitive) attributes such as race, gender, and age. A prerequisite for existing methods to achieve counterfactual fairness is the prior human knowledge of the causal model for the data. In this work, we address the problem of counterfactually fair prediction from observational data without given causal models by proposing a novel framework CLAIRE.
  arXiv  Detail & Related papers  (2023-07-17T04:08:29Z)
• Guide the Learner: Controlling Product of Experts Debiasing Method Based on Token Attribution Similarities [17.082695183953486]
  A popular workaround is to train a robust model by re-weighting training examples based on a secondary biased model. Here, the underlying assumption is that the biased model resorts to shortcut features. We introduce a fine-tuning strategy that incorporates the similarity between the main and biased model attribution scores in a Product of Experts loss function.
  arXiv  Detail & Related papers  (2023-02-06T15:21:41Z)
• Biases in Inverse Ising Estimates of Near-Critical Behaviour [0.0]
  Inverse inference allows pairwise interactions to be reconstructed from empirical correlations. We show that estimators used for this inference, such as Pseudo-likelihood (PLM), are biased. Data-driven methods are explored and applied to a functional magnetic resonance imaging (fMRI) dataset from neuroscience.
  arXiv  Detail & Related papers  (2023-01-13T14:01:43Z)
• Bias-inducing geometries: an exactly solvable data model with fairness implications [12.532003449620607]
  We introduce an exactly solvable high-dimensional model of data imbalance. We analytically unpack the typical properties of learning models trained in this synthetic framework. We obtain exact predictions for the observables that are commonly employed for fairness assessment.
  arXiv  Detail & Related papers  (2022-05-31T16:27:57Z)
• General Greedy De-bias Learning [163.65789778416172]
  We propose a General Greedy De-bias learning framework (GGD), which greedily trains the biased models and the base model like gradient descent in functional space. GGD can learn a more robust base model under the settings of both task-specific biased models with prior knowledge and self-ensemble biased model without prior knowledge.
  arXiv  Detail & Related papers  (2021-12-20T14:47:32Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• Loss Estimators Improve Model Generalization [36.520569284970456]
  We propose to train a loss estimator alongside the predictive model, using a contrastive training objective, to directly estimate the prediction uncertainties. We show the impact of loss estimators on model generalization, in terms of both its fidelity on in-distribution data and its ability to detect out of distribution samples or new classes unseen during training.
  arXiv  Detail & Related papers  (2021-03-05T16:35:10Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance [70.31427277842239]
  We introduce a novel debiasing method called confidence regularization. It discourages models from exploiting biases while enabling them to receive enough incentive to learn from all the training examples. We evaluate our method on three NLU tasks and show that, in contrast to its predecessors, it improves the performance on out-of-distribution datasets.
  arXiv  Detail & Related papers  (2020-05-01T11:22:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.