A First Look at Fairness of Machine Learning Based Code Reviewer Recommendation

• URL: http://arxiv.org/abs/2307.11298v1
• Date: Fri, 21 Jul 2023 01:57:51 GMT
• Title: A First Look at Fairness of Machine Learning Based Code Reviewer Recommendation
• Authors: Mohammad Mahdi Mohajer, Alvine Boaye Belle, Nima Shiri harzevili, Junjie Wang, Hadi Hemmati, Song Wang, Zhen Ming (Jack) Jiang
• Abstract summary: This paper conducts the first study toward investigating the issue of fairness of ML applications in the software engineering (SE) domain. Our empirical study demonstrates that current state-of-the-art ML-based code reviewer recommendation techniques exhibit unfairness and discriminating behaviors. This paper also discusses the reasons why the studied ML-based code reviewer recommendation systems are unfair and provides solutions to mitigate the unfairness.
• Score: 14.50773969815661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The fairness of machine learning (ML) approaches is critical to the reliability of modern artificial intelligence systems. Despite extensive study on this topic, the fairness of ML models in the software engineering (SE) domain has not been well explored yet. As a result, many ML-powered software systems, particularly those utilized in the software engineering community, continue to be prone to fairness issues. Taking one of the typical SE tasks, i.e., code reviewer recommendation, as a subject, this paper conducts the first study toward investigating the issue of fairness of ML applications in the SE domain. Our empirical study demonstrates that current state-of-the-art ML-based code reviewer recommendation techniques exhibit unfairness and discriminating behaviors. Specifically, male reviewers get on average 7.25% more recommendations than female code reviewers compared to their distribution in the reviewer set. This paper also discusses the reasons why the studied ML-based code reviewer recommendation systems are unfair and provides solutions to mitigate the unfairness. Our study further indicates that the existing mitigation methods can enhance fairness by 100% in projects with a similar distribution of protected and privileged groups, but their effectiveness in improving fairness on imbalanced or skewed data is limited. Eventually, we suggest a solution to overcome the drawbacks of existing mitigation techniques and tackle bias in datasets that are imbalanced or skewed.

Related papers

• Estimating Fr\'echet bounds for validating programmatic weak supervision [50.13475056199486]
  We develop methods for estimating Fr'eche's bounds on (possibly high-dimensional) distribution classes in which some variables are continuous-valued. We demonstrate the usefulness of our algorithms by evaluating the performance of machine learning (ML) models trained with programmatic weak supervision (PWS)
  arXiv  Detail & Related papers  (2023-12-07T07:15:11Z)
• Fair Few-shot Learning with Auxiliary Sets [53.30014767684218]
  In many machine learning (ML) tasks, only very few labeled data samples can be collected, which can lead to inferior fairness performance. In this paper, we define the fairness-aware learning task with limited training samples as the emphfair few-shot learning problem. We devise a novel framework that accumulates fairness-aware knowledge across different meta-training tasks and then generalizes the learned knowledge to meta-test tasks.
  arXiv  Detail & Related papers  (2023-08-28T06:31:37Z)
• Assessing Perceived Fairness from Machine Learning Developer's Perspective [0.0]
  unfairness is triggered due to bias in the data, curation process, erroneous assumptions, and implicit bias rendered within the algorithmic development process. In particular, ML developers have not been the focus of research relating to perceived fairness. This paper performs an exploratory pilot study to assess the attributes of this construct using a systematic focus group of developers.
  arXiv  Detail & Related papers  (2023-04-07T17:30:37Z)
• Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking [16.196269707571904]
  We present a novel counterfactual approach to tackle the root causes of bias in Machine Learning software. Our approach combines models optimized for both performance and fairness, resulting in an optimal solution in both aspects.
  arXiv  Detail & Related papers  (2023-02-16T01:27:26Z)
• Improving Recommendation Fairness via Data Augmentation [66.4071365614835]
  Collaborative filtering based recommendation learns users' preferences from all users' historical behavior data, and has been popular to facilitate decision making. A recommender system is considered unfair when it does not perform equally well for different user groups according to users' sensitive attributes. In this paper, we study how to improve recommendation fairness from the data augmentation perspective.
  arXiv  Detail & Related papers  (2023-02-13T13:11:46Z)
• Evaluating Machine Unlearning via Epistemic Uncertainty [78.27542864367821]
  This work presents an evaluation of Machine Unlearning algorithms based on uncertainty. This is the first definition of a general evaluation of our best knowledge.
  arXiv  Detail & Related papers  (2022-08-23T09:37:31Z)
• D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias [57.87117733071416]
  We propose D-BIAS, a visual interactive tool that embodies human-in-the-loop AI approach for auditing and mitigating social biases. A user can detect the presence of bias against a group by identifying unfair causal relationships in the causal network. For each interaction, say weakening/deleting a biased causal edge, the system uses a novel method to simulate a new (debiased) dataset.
  arXiv  Detail & Related papers  (2022-08-10T03:41:48Z)
• A Framework for Fairness: A Systematic Review of Existing Fair AI Solutions [4.594159253008448]
  A large portion of fairness research has gone to producing tools that machine learning practitioners can use to audit for bias while designing their algorithms. There is a lack of application of these fairness solutions in practice. This review provides an in-depth summary of the algorithmic bias issues that have been defined and the fairness solution space that has been proposed.
  arXiv  Detail & Related papers  (2021-12-10T17:51:20Z)
• Algorithmic Factors Influencing Bias in Machine Learning [2.055949720959582]
  We show how irreducible error, regularization and feature and class imbalance can contribute to this underestimation. The paper concludes with a demonstration of how the careful management of synthetic counterfactuals can ameliorate the impact of this underestimation bias.
  arXiv  Detail & Related papers  (2021-04-28T20:45:41Z)
• Fair Representation Learning for Heterogeneous Information Networks [35.80367469624887]
  We propose a comprehensive set of de-biasing methods for fair HINs representation learning. We study the behavior of these algorithms, especially their capability in balancing the trade-off between fairness and prediction accuracy. We evaluate the performance of the proposed methods in an automated career counseling application.
  arXiv  Detail & Related papers  (2021-04-18T08:28:18Z)
• BeFair: Addressing Fairness in the Banking Sector [54.08949958349055]
  We present the initial results of an industrial open innovation project in the banking sector. We propose a general roadmap for fairness in ML and the implementation of a toolkit called BeFair that helps to identify and mitigate bias.
  arXiv  Detail & Related papers  (2021-02-03T16:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.