How fair are we? From conceptualization to automated assessment of fairness definitions

• URL: http://arxiv.org/abs/2404.09919v1
• Date: Mon, 15 Apr 2024 16:46:17 GMT
• Title: How fair are we? From conceptualization to automated assessment of fairness definitions
• Authors: Giordano d'Aloisio, Claudio Di Sipio, Antinisca Di Marco, Davide Di Ruscio,
• Abstract summary: MODNESS is a model-driven approach for user-defined fairness concepts in software systems. It generates the source code to implement fair assessment based on these custom definitions. Our findings reveal that most of the current approaches do not support user-defined fairness concepts.
• Score: 6.741000368514124
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Fairness is a critical concept in ethics and social domains, but it is also a challenging property to engineer in software systems. With the increasing use of machine learning in software systems, researchers have been developing techniques to automatically assess the fairness of software systems. Nonetheless, a significant proportion of these techniques rely upon pre-established fairness definitions, metrics, and criteria, which may fail to encompass the wide-ranging needs and preferences of users and stakeholders. To overcome this limitation, we propose a novel approach, called MODNESS, that enables users to customize and define their fairness concepts using a dedicated modeling environment. Our approach guides the user through the definition of new fairness concepts also in emerging domains, and the specification and composition of metrics for its evaluation. Ultimately, MODNESS generates the source code to implement fair assessment based on these custom definitions. In addition, we elucidate the process we followed to collect and analyze relevant literature on fairness assessment in software engineering (SE). We compare MODNESS with the selected approaches and evaluate how they support the distinguishing features identified by our study. Our findings reveal that i) most of the current approaches do not support user-defined fairness concepts; ii) our approach can cover two additional application domains not addressed by currently available tools, i.e., mitigating bias in recommender systems for software engineering and Arduino software component recommendations; iii) MODNESS demonstrates the capability to overcome the limitations of the only two other Model-Driven Engineering-based approaches for fairness assessment.

Related papers

• Pessimistic Evaluation [58.736490198613154]
  We argue that evaluating information access systems assumes utilitarian values not aligned with traditions of information access based on equal access. We advocate for pessimistic evaluation of information access systems focusing on worst case utility.
  arXiv  Detail & Related papers  (2024-10-17T15:40:09Z)
• Benchmarks as Microscopes: A Call for Model Metrology [76.64402390208576]
  Modern language models (LMs) pose a new challenge in capability assessment. To be confident in our metrics, we need a new discipline of model metrology.
  arXiv  Detail & Related papers  (2024-07-22T17:52:12Z)
• A Benchmark for Fairness-Aware Graph Learning [58.515305543487386]
  We present an extensive benchmark on ten representative fairness-aware graph learning methods. Our in-depth analysis reveals key insights into the strengths and limitations of existing methods.
  arXiv  Detail & Related papers  (2024-07-16T18:43:43Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making. We present a process-based benchmark MR-Ben that demands a meta-reasoning skill. Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• An Audit Framework for Technical Assessment of Binary Classifiers [0.0]
  Multilevel models using logistic regression (MLogRM) and random forest models (RFM) are increasingly deployed in industry for the purpose of binary classification. The European Commission's proposed Artificial Intelligence Act (AIA) necessitates, under certain conditions, that application of such models is fair, transparent, and ethical. This paper proposes and demonstrates an audit framework for technical assessment of RFMs and MLogRMs by focussing on model-, discrimination, and transparency & explainability-related aspects.
  arXiv  Detail & Related papers  (2022-11-17T12:48:11Z)
• Experiments on Generalizability of User-Oriented Fairness in Recommender Systems [2.0932879442844476]
  A fairness-aware recommender system aims to treat different user groups similarly. We propose a user-centered fairness re-ranking framework applied on top of a base ranking model. We evaluate the final recommendations provided by the re-ranking framework from both user- (e.g., NDCG) and item-side (e.g., novelty, item-fairness) metrics.
  arXiv  Detail & Related papers  (2022-05-17T12:36:30Z)
• fairlib: A Unified Framework for Assessing and Improving Classification Fairness [66.27822109651757]
  fairlib is an open-source framework for assessing and improving classification fairness. We implement 14 debiasing methods, including pre-processing, at-training-time, and post-processing approaches. The built-in metrics cover the most commonly used fairness criterion and can be further generalized and customized for fairness evaluation.
  arXiv  Detail & Related papers  (2022-05-04T03:50:23Z)
• Fair Representation Learning for Heterogeneous Information Networks [35.80367469624887]
  We propose a comprehensive set of de-biasing methods for fair HINs representation learning. We study the behavior of these algorithms, especially their capability in balancing the trade-off between fairness and prediction accuracy. We evaluate the performance of the proposed methods in an automated career counseling application.
  arXiv  Detail & Related papers  (2021-04-18T08:28:18Z)
• Getting Fairness Right: Towards a Toolbox for Practitioners [2.4364387374267427]
  The potential risk of AI systems unintentionally embedding and reproducing bias has attracted the attention of machine learning practitioners and society at large. This paper proposes to draft a toolbox which helps practitioners to ensure fair AI practices.
  arXiv  Detail & Related papers  (2020-03-15T20:53:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.