FairX: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability

• URL: http://arxiv.org/abs/2406.14281v4
• Date: Tue, 3 Sep 2024 12:38:22 GMT
• Title: FairX: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability
• Authors: Md Fahim Sikder, Resmi Ramachandranpillai, Daniel de Leng, Fredrik Heintz,
• Abstract summary: We present FairX, an open-source benchmarking tool designed for the comprehensive analysis of models under the umbrella of fairness, utility, and eXplainability (XAI) FairX enables users to train benchmarking bias-mitigation models and evaluate their fairness using a wide array of fairness metrics, data utility metrics, and generate explanations for model predictions, all within a unified framework.
• Score: 4.1942958779358674
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present FairX, an open-source Python-based benchmarking tool designed for the comprehensive analysis of models under the umbrella of fairness, utility, and eXplainability (XAI). FairX enables users to train benchmarking bias-mitigation models and evaluate their fairness using a wide array of fairness metrics, data utility metrics, and generate explanations for model predictions, all within a unified framework. Existing benchmarking tools do not have the way to evaluate synthetic data generated from fair generative models, also they do not have the support for training fair generative models either. In FairX, we add fair generative models in the collection of our fair-model library (pre-processing, in-processing, post-processing) and evaluation metrics for evaluating the quality of synthetic fair data. This version of FairX supports both tabular and image datasets. It also allows users to provide their own custom datasets. The open-source FairX benchmarking package is publicly available at \url{https://github.com/fahim-sikder/FairX}.

Related papers

• LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content [62.816876067499415]
  We propose LiveXiv: a scalable evolving live benchmark based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate visual question-answer pairs. We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models true abilities.
  arXiv  Detail & Related papers  (2024-10-14T17:51:23Z)
• Benchmarking the Fairness of Image Upsampling Methods [29.01986714656294]
  We develop a set of metrics for performance and fairness of conditional generative models. We benchmark their imbalances and diversity. As part of the study, a subset of datasets replicates the racial distribution of common-scale face.
  arXiv  Detail & Related papers  (2024-01-24T16:13:26Z)
• FairGridSearch: A Framework to Compare Fairness-Enhancing Models [0.0]
  This paper focuses on binary classification and proposes FairGridSearch, a novel framework for comparing fairness-enhancing models. The study applies FairGridSearch to three popular datasets (Adult, COMPAS, and German Credit) and analyzes the impacts of metric selection, base estimator choice, and classification threshold on model fairness.
  arXiv  Detail & Related papers  (2024-01-04T10:29:02Z)
• FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods [84.1077756698332]
  This paper introduces the Fair Fairness Benchmark (textsfFFB), a benchmarking framework for in-processing group fairness methods. We provide a comprehensive analysis of state-of-the-art methods to ensure different notions of group fairness.
  arXiv  Detail & Related papers  (2023-06-15T19:51:28Z)
• DualFair: Fair Representation Learning at Both Group and Individual Levels via Contrastive Self-supervision [73.80009454050858]
  This work presents a self-supervised model, called DualFair, that can debias sensitive attributes like gender and race from learned representations. Our model jointly optimize for two fairness criteria - group fairness and counterfactual fairness.
  arXiv  Detail & Related papers  (2023-03-15T07:13:54Z)
• Towards Algorithmic Fairness by means of Instance-level Data Re-weighting based on Shapley Values [6.752538702870792]
  We propose FairShap, a novel instance-level data re-weighting method for fair algorithmic decision-making. We empirically validate FairShap on several state-of-the-art datasets of different nature. We show how it yields fairer models with similar levels of accuracy than the baselines.
  arXiv  Detail & Related papers  (2023-03-03T13:53:36Z)
• FairPy: A Toolkit for Evaluation of Prediction Biases and their Mitigation in Large Language Models [12.62204775625353]
  Recent studies have demonstrated that large pretrained language models (LLMs) such as BERT and GPT-2 exhibit biases in token prediction. We present a comprehensive survey of such techniques tailored towards widely used LLMs such as BERT, GPT-2, etc. We additionally introduce Fairpy, a modular and toolkit that provides plug-and-play interfaces for integrating these mathematical tools.
  arXiv  Detail & Related papers  (2023-02-10T20:54:10Z)
• fairlib: A Unified Framework for Assessing and Improving Classification Fairness [66.27822109651757]
  fairlib is an open-source framework for assessing and improving classification fairness. We implement 14 debiasing methods, including pre-processing, at-training-time, and post-processing approaches. The built-in metrics cover the most commonly used fairness criterion and can be further generalized and customized for fairness evaluation.
  arXiv  Detail & Related papers  (2022-05-04T03:50:23Z)
• DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks [71.6879432974126]
  We introduce DECAF: a GAN-based fair synthetic data generator for tabular data. We show that DECAF successfully removes undesired bias and is capable of generating high-quality synthetic data. We provide theoretical guarantees on the generator's convergence and the fairness of downstream models.
  arXiv  Detail & Related papers  (2021-10-25T12:39:56Z)
• Improving Label Quality by Jointly Modeling Items and Annotators [68.8204255655161]
  We propose a fully Bayesian framework for learning ground truth labels from noisy annotators. Our framework ensures scalability by factoring a generative, Bayesian soft clustering model over label distributions into the classic David and Skene joint annotator-data model.
  arXiv  Detail & Related papers  (2021-06-20T02:15:20Z)
• Group Fairness by Probabilistic Modeling with Latent Fair Decisions [36.20281545470954]
  This paper studies learning fair probability distributions from biased data by explicitly modeling a latent variable that represents a hidden, unbiased label. We aim to achieve demographic parity by enforcing certain independencies in the learned model. We also show that group fairness guarantees are meaningful only if the distribution used to provide those guarantees indeed captures the real-world data.
  arXiv  Detail & Related papers  (2020-09-18T19:13:23Z)
• Fairness by Explicability and Adversarial SHAP Learning [0.0]
  We propose a new definition of fairness that emphasises the role of an external auditor and model explicability. We develop a framework for mitigating model bias using regularizations constructed from the SHAP values of an adversarial surrogate model. We demonstrate our approaches using gradient and adaptive boosting on: a synthetic dataset, the UCI Adult (Census) dataset and a real-world credit scoring dataset.
  arXiv  Detail & Related papers  (2020-03-11T14:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.