A First Look at Fairness of Machine Learning Based Code Reviewer Recommendation

• URL: http://arxiv.org/abs/2307.11298v1
• Date: Fri, 21 Jul 2023 01:57:51 GMT
• Title: A First Look at Fairness of Machine Learning Based Code Reviewer Recommendation
• Authors: Mohammad Mahdi Mohajer, Alvine Boaye Belle, Nima Shiri harzevili, Junjie Wang, Hadi Hemmati, Song Wang, Zhen Ming (Jack) Jiang
• Abstract summary: This paper conducts the first study toward investigating the issue of fairness of ML applications in the software engineering (SE) domain. Our empirical study demonstrates that current state-of-the-art ML-based code reviewer recommendation techniques exhibit unfairness and discriminating behaviors. This paper also discusses the reasons why the studied ML-based code reviewer recommendation systems are unfair and provides solutions to mitigate the unfairness.
• Score: 14.50773969815661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The fairness of machine learning (ML) approaches is critical to the reliability of modern artificial intelligence systems. Despite extensive study on this topic, the fairness of ML models in the software engineering (SE) domain has not been well explored yet. As a result, many ML-powered software systems, particularly those utilized in the software engineering community, continue to be prone to fairness issues. Taking one of the typical SE tasks, i.e., code reviewer recommendation, as a subject, this paper conducts the first study toward investigating the issue of fairness of ML applications in the SE domain. Our empirical study demonstrates that current state-of-the-art ML-based code reviewer recommendation techniques exhibit unfairness and discriminating behaviors. Specifically, male reviewers get on average 7.25% more recommendations than female code reviewers compared to their distribution in the reviewer set. This paper also discusses the reasons why the studied ML-based code reviewer recommendation systems are unfair and provides solutions to mitigate the unfairness. Our study further indicates that the existing mitigation methods can enhance fairness by 100% in projects with a similar distribution of protected and privileged groups, but their effectiveness in improving fairness on imbalanced or skewed data is limited. Eventually, we suggest a solution to overcome the drawbacks of existing mitigation techniques and tackle bias in datasets that are imbalanced or skewed.

Related papers

• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge [84.34545223897578]
  Despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. We identify 12 key potential biases and propose a new automated bias quantification framework-CALM- which quantifies and analyzes each type of bias in LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.
  arXiv  Detail & Related papers  (2024-10-03T17:53:30Z)
• GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models [73.23743278545321]
  Large language models (LLMs) have exhibited remarkable capabilities in natural language generation, but have also been observed to magnify societal biases. GenderCARE is a comprehensive framework that encompasses innovative Criteria, bias Assessment, Reduction techniques, and Evaluation metrics.
  arXiv  Detail & Related papers  (2024-08-22T15:35:46Z)
• A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender System [9.470545149911072]
  This paper proposes a normative framework to benchmark consumer fairness in LLM-powered recommender systems. We argue that this gap can lead to arbitrary conclusions about fairness. Experiments on the MovieLens dataset on consumer fairness reveal fairness deviations in age-based recommendations.
  arXiv  Detail & Related papers  (2024-05-03T16:25:27Z)
• Fair Few-shot Learning with Auxiliary Sets [53.30014767684218]
  In many machine learning (ML) tasks, only very few labeled data samples can be collected, which can lead to inferior fairness performance. In this paper, we define the fairness-aware learning task with limited training samples as the emphfair few-shot learning problem. We devise a novel framework that accumulates fairness-aware knowledge across different meta-training tasks and then generalizes the learned knowledge to meta-test tasks.
  arXiv  Detail & Related papers  (2023-08-28T06:31:37Z)
• Assessing Perceived Fairness from Machine Learning Developer's Perspective [0.0]
  unfairness is triggered due to bias in the data, curation process, erroneous assumptions, and implicit bias rendered within the algorithmic development process. In particular, ML developers have not been the focus of research relating to perceived fairness. This paper performs an exploratory pilot study to assess the attributes of this construct using a systematic focus group of developers.
  arXiv  Detail & Related papers  (2023-04-07T17:30:37Z)
• Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking [16.196269707571904]
  We present a novel counterfactual approach to tackle the root causes of bias in Machine Learning software. Our approach combines models optimized for both performance and fairness, resulting in an optimal solution in both aspects.
  arXiv  Detail & Related papers  (2023-02-16T01:27:26Z)
• Improving Recommendation Fairness via Data Augmentation [66.4071365614835]
  Collaborative filtering based recommendation learns users' preferences from all users' historical behavior data, and has been popular to facilitate decision making. A recommender system is considered unfair when it does not perform equally well for different user groups according to users' sensitive attributes. In this paper, we study how to improve recommendation fairness from the data augmentation perspective.
  arXiv  Detail & Related papers  (2023-02-13T13:11:46Z)
• D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias [57.87117733071416]
  We propose D-BIAS, a visual interactive tool that embodies human-in-the-loop AI approach for auditing and mitigating social biases. A user can detect the presence of bias against a group by identifying unfair causal relationships in the causal network. For each interaction, say weakening/deleting a biased causal edge, the system uses a novel method to simulate a new (debiased) dataset.
  arXiv  Detail & Related papers  (2022-08-10T03:41:48Z)
• Algorithmic Factors Influencing Bias in Machine Learning [2.055949720959582]
  We show how irreducible error, regularization and feature and class imbalance can contribute to this underestimation. The paper concludes with a demonstration of how the careful management of synthetic counterfactuals can ameliorate the impact of this underestimation bias.
  arXiv  Detail & Related papers  (2021-04-28T20:45:41Z)
• Fair Representation Learning for Heterogeneous Information Networks [35.80367469624887]
  We propose a comprehensive set of de-biasing methods for fair HINs representation learning. We study the behavior of these algorithms, especially their capability in balancing the trade-off between fairness and prediction accuracy. We evaluate the performance of the proposed methods in an automated career counseling application.
  arXiv  Detail & Related papers  (2021-04-18T08:28:18Z)
• BeFair: Addressing Fairness in the Banking Sector [54.08949958349055]
  We present the initial results of an industrial open innovation project in the banking sector. We propose a general roadmap for fairness in ML and the implementation of a toolkit called BeFair that helps to identify and mitigate bias.
  arXiv  Detail & Related papers  (2021-02-03T16:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.