FairRec: Fairness Testing for Deep Recommender Systems
- URL: http://arxiv.org/abs/2304.07030v1
- Date: Fri, 14 Apr 2023 09:49:55 GMT
- Title: FairRec: Fairness Testing for Deep Recommender Systems
- Authors: Huizhong Guo, Jinfeng Li, Jingyi Wang, Xiangyu Liu, Dongxia Wang,
Zehong Hu, Rong Zhang and Hui Xue
- Abstract summary: We propose a unified framework that supports fairness testing of deep learning-based recommender systems.
We also propose a novel, efficient search-based testing approach to tackle the new challenge.
- Score: 21.420524191767335
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep learning-based recommender systems (DRSs) are increasingly and widely
deployed in the industry, which brings significant convenience to people's
daily life in different ways. However, recommender systems are also shown to
suffer from multiple issues,e.g., the echo chamber and the Matthew effect, of
which the notation of "fairness" plays a core role.While many fairness
notations and corresponding fairness testing approaches have been developed for
traditional deep classification models, they are essentially hardly applicable
to DRSs. One major difficulty is that there still lacks a systematic
understanding and mapping between the existing fairness notations and the
diverse testing requirements for deep recommender systems, not to mention
further testing or debugging activities. To address the gap, we propose
FairRec, a unified framework that supports fairness testing of DRSs from
multiple customized perspectives, e.g., model utility, item diversity, item
popularity, etc. We also propose a novel, efficient search-based testing
approach to tackle the new challenge, i.e., double-ended discrete particle
swarm optimization (DPSO) algorithm, to effectively search for hidden fairness
issues in the form of certain disadvantaged groups from a vast number of
candidate groups. Given the testing report, by adopting a simple re-ranking
mitigation strategy on these identified disadvantaged groups, we show that the
fairness of DRSs can be significantly improved. We conducted extensive
experiments on multiple industry-level DRSs adopted by leading companies. The
results confirm that FairRec is effective and efficient in identifying the
deeply hidden fairness issues, e.g., achieving 95% testing accuracy with half
to 1/8 time.
Related papers
- Thinking Racial Bias in Fair Forgery Detection: Models, Datasets and Evaluations [63.52709761339949]
We first contribute a dedicated dataset called the Fair Forgery Detection (FairFD) dataset, where we prove the racial bias of public state-of-the-art (SOTA) methods.
We design novel metrics including Approach Averaged Metric and Utility Regularized Metric, which can avoid deceptive results.
We also present an effective and robust post-processing technique, Bias Pruning with Fair Activations (BPFA), which improves fairness without requiring retraining or weight updates.
arXiv Detail & Related papers (2024-07-19T14:53:18Z) - A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender System [9.470545149911072]
This paper proposes a normative framework to benchmark consumer fairness in LLM-powered recommender systems.
We argue that this gap can lead to arbitrary conclusions about fairness.
Experiments on the MovieLens dataset on consumer fairness reveal fairness deviations in age-based recommendations.
arXiv Detail & Related papers (2024-05-03T16:25:27Z) - Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction [56.17020601803071]
Recent research shows that pre-trained language models (PLMs) suffer from "prompt bias" in factual knowledge extraction.
This paper aims to improve the reliability of existing benchmarks by thoroughly investigating and mitigating prompt bias.
arXiv Detail & Related papers (2024-03-15T02:04:35Z) - Understanding Biases in ChatGPT-based Recommender Systems: Provider Fairness, Temporal Stability, and Recency [9.882829614199453]
This paper explores the biases in ChatGPT-based recommender systems, focusing on provider fairness (item-side fairness)
In the first experiment, we assess seven distinct prompt scenarios on top-K recommendation accuracy and fairness.
Embedding fairness into system roles, such as "act as a fair recommender," proved more effective than fairness directives within prompts.
arXiv Detail & Related papers (2024-01-19T08:09:20Z) - Fair-CDA: Continuous and Directional Augmentation for Group Fairness [48.84385689186208]
We propose a fine-grained data augmentation strategy for imposing fairness constraints.
We show that group fairness can be achieved by regularizing the models on transition paths of sensitive features between groups.
Our proposed method does not assume any data generative model and ensures good generalization for both accuracy and fairness.
arXiv Detail & Related papers (2023-04-01T11:23:00Z) - Equal Improvability: A New Fairness Notion Considering the Long-term
Impact [27.72859815965265]
We propose a new fairness notion called Equal Improvability (EI)
EI equalizes the potential acceptance rate of the rejected samples across different groups.
We show that proposed EI-regularized algorithms encourage us to find a fair classifier in terms of EI.
arXiv Detail & Related papers (2022-10-13T04:59:28Z) - Error Parity Fairness: Testing for Group Fairness in Regression Tasks [5.076419064097733]
This work presents error parity as a regression fairness notion and introduces a testing methodology to assess group fairness.
It is followed by a suitable permutation test to compare groups on several statistics to explore disparities and identify impacted groups.
Overall, the proposed regression fairness testing methodology fills a gap in the fair machine learning literature and may serve as a part of larger accountability assessments and algorithm audits.
arXiv Detail & Related papers (2022-08-16T17:47:20Z) - Balancing Accuracy and Fairness for Interactive Recommendation with
Reinforcement Learning [68.25805655688876]
Fairness in recommendation has attracted increasing attention due to bias and discrimination possibly caused by traditional recommenders.
We propose a reinforcement learning based framework, FairRec, to dynamically maintain a long-term balance between accuracy and fairness in IRS.
Extensive experiments validate that FairRec can improve fairness, while preserving good recommendation quality.
arXiv Detail & Related papers (2021-06-25T02:02:51Z) - Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking
Fairness and Algorithm Utility [54.179859639868646]
Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data.
There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups.
We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
arXiv Detail & Related papers (2020-06-15T10:08:39Z) - Contrastive Learning for Debiased Candidate Generation in Large-Scale
Recommender Systems [84.3996727203154]
We show that a popular choice of contrastive loss is equivalent to reducing the exposure bias via inverse propensity weighting.
We further improve upon CLRec and propose Multi-CLRec, for accurate multi-intention aware bias reduction.
Our methods have been successfully deployed in Taobao, where at least four-month online A/B tests and offline analyses demonstrate its substantial improvements.
arXiv Detail & Related papers (2020-05-20T08:15:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.