On the Consistency of Fairness Measurement Methods for Regression Tasks

• URL: http://arxiv.org/abs/2406.13681v1
• Date: Wed, 19 Jun 2024 16:35:23 GMT
• Title: On the Consistency of Fairness Measurement Methods for Regression Tasks
• Authors: Abdalwahab Almajed, Maryam Tabar, Peyman Najafirad,
• Abstract summary: This paper studies the consistency of the output of various fairness measurement methods. It finds that while some fairness measurement approaches show strong consistency across various regression tasks, certain methods show a relatively poor consistency in certain regression tasks.
• Score: 4.4701415309453285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With growing applications of Machine Learning (ML) techniques in the real world, it is highly important to ensure that these models work in an equitable manner. One main step in ensuring fairness is to effectively measure fairness, and to this end, various metrics have been proposed in the past literature. While the computation of those metrics are straightforward in the classification set-up, it is computationally intractable in the regression domain. To address the challenge of computational intractability, past literature proposed various methods to approximate such metrics. However, they did not verify the extent to which the output of such approximation algorithms are consistent with each other. To fill this gap, this paper comprehensively studies the consistency of the output of various fairness measurement methods through conducting an extensive set of experiments on various regression tasks. As a result, it finds that while some fairness measurement approaches show strong consistency across various regression tasks, certain methods show a relatively poor consistency in certain regression tasks. This, in turn, calls for a more principled approach for measuring fairness in the regression domain.

Related papers

• Boosting methods for interval-censored data with regression and classification [4.5041160025507585]
  We introduce novel non-parametric boosting methods for regression and classification tasks with interval-censored data.<n>Our approaches leverage censoring unbiased transformations to adjust loss functions and impute transformed responses.<n>We rigorously establish their theoretical properties, including optimality and mean squared error trade-offs.
  arXiv  Detail & Related papers  (2026-01-25T20:05:57Z)
• A Comprehensive Evaluation of the Sensitivity of Density-Ratio Estimation Based Fairness Measurement in Regression [4.4701415309453285]
  This paper develops a set of fairness measurement methods with various density-ratio estimation cores.<n>Experiments show that the choice of density-ratio estimation core could significantly affect the outcome of fairness measurement method.<n>These observations suggest major issues with density-ratio estimation based fairness measurement in regression.
  arXiv  Detail & Related papers  (2025-08-20T09:54:55Z)
• Distances for Markov chains from sample streams [16.443304244634767]
  Bisimulation metrics are powerful tools for measuring similarities between processes, and specifically Markov chains.<n>Recent advances have uncovered that bisimulation metrics are, in fact, optimal-transport distances, which has enabled the development of fast algorithms for computing such metrics with provable accuracy and runtime guarantees.<n>This is often an impractical assumption in most real-world scenarios, where typically only sample trajectories are available.<n>We propose a new optimization method that addresses this limitation and estimates bisimulation metrics based on sample access, without requiring explicit transition models.
  arXiv  Detail & Related papers  (2025-05-23T15:09:04Z)
• Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Sampling [14.668634411361307]
  We introduce a benchmark that evaluates sampling methods using a standardized task suite and a broad range of performance criteria. We study existing metrics for quantifying mode collapse and introduce novel metrics for this purpose.
  arXiv  Detail & Related papers  (2024-06-11T16:23:33Z)
• Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement [69.51496713076253]
  In this paper, we focus on the aforementioned efficiency aspects of existing MTL methods. We first carry out large-scale experiments of the methods with smaller backbones and on a the MetaGraspNet dataset as a new test ground. We also propose Feature Disentanglement measure as a novel and efficient identifier of the challenges in MTL.
  arXiv  Detail & Related papers  (2024-02-05T22:15:55Z)
• Medoid splits for efficient random forests in metric spaces [0.0]
  This paper revisits an adaptation of the random forest for Fr'echet regression, addressing the challenge of regression in metric spaces. We introduce a new splitting rule that circumvents the computationally expensive operation of Fr'echet means by substituting with a medoid-based approach.
  arXiv  Detail & Related papers  (2023-06-29T15:32:11Z)
• SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum Cocoercive Variational Inequalities [137.6408511310322]
  We consider the problem of finite-sum cocoercive variational inequalities. For strongly monotone problems it is possible to achieve linear convergence to a solution using this method.
  arXiv  Detail & Related papers  (2022-10-12T08:04:48Z)
• Robust online joint state/input/parameter estimation of linear systems [0.0]
  This paper presents a method for jointly estimating the state, input, and parameters of linear systems in an online fashion. The method is specially designed for measurements that are corrupted with non-Gaussian noise or outliers.
  arXiv  Detail & Related papers  (2022-04-12T09:41:28Z)
• On Modality Bias Recognition and Reduction [70.69194431713825]
  We study the modality bias problem in the context of multi-modal classification. We propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned. Our method yields remarkable performance improvements compared with the baselines.
  arXiv  Detail & Related papers  (2022-02-25T13:47:09Z)
• Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions [91.63716984911278]
  We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
  arXiv  Detail & Related papers  (2021-11-11T14:28:12Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• Towards Certified Robustness of Distance Metric Learning [53.96113074344632]
  We advocate imposing an adversarial margin in the input space so as to improve the generalization and robustness of metric learning algorithms. We show that the enlarged margin is beneficial to the generalization ability by using the theoretical technique of algorithmic robustness.
  arXiv  Detail & Related papers  (2020-06-10T16:51:53Z)
• Fair inference on error-prone outcomes [0.0]
  We show that, when an error-prone proxy target is used, existing methods to assess and calibrate fairness criteria do not extend to the true target variable of interest. We suggest a framework resulting from the combination of fair ML methods and measurement models found in the statistical literature. In a healthcare decision problem, we find that using a latent variable model to account for measurement error removes the unfairness detected previously.
  arXiv  Detail & Related papers  (2020-03-17T10:31:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.