A Large-Scale Empirical Study on Improving the Fairness of Image Classification Models

• URL: http://arxiv.org/abs/2401.03695v2
• Date: Sun, 24 Mar 2024 02:03:55 GMT
• Title: A Large-Scale Empirical Study on Improving the Fairness of Image Classification Models
• Authors: Junjie Yang, Jiajun Jiang, Zeyu Sun, Junjie Chen,
• Abstract summary: This paper conducts the first large-scale empirical study to compare the performance of existing state-of-the-art fairness improving techniques. Our findings reveal substantial variations in the performance of each method across different datasets and sensitive attributes. Different fairness evaluation metrics, due to their distinct focuses, yield significantly different assessment results.
• Score: 22.522156479335706
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fairness has been a critical issue that affects the adoption of deep learning models in real practice. To improve model fairness, many existing methods have been proposed and evaluated to be effective in their own contexts. However, there is still no systematic evaluation among them for a comprehensive comparison under the same context, which makes it hard to understand the performance distinction among them, hindering the research progress and practical adoption of them. To fill this gap, this paper endeavours to conduct the first large-scale empirical study to comprehensively compare the performance of existing state-of-the-art fairness improving techniques. Specifically, we target the widely-used application scenario of image classification, and utilized three different datasets and five commonly-used performance metrics to assess in total 13 methods from diverse categories. Our findings reveal substantial variations in the performance of each method across different datasets and sensitive attributes, indicating over-fitting on specific datasets by many existing methods. Furthermore, different fairness evaluation metrics, due to their distinct focuses, yield significantly different assessment results. Overall, we observe that pre-processing methods and in-processing methods outperform post-processing methods, with pre-processing methods exhibiting the best performance. Our empirical study offers comprehensive recommendations for enhancing fairness in deep learning models. We approach the problem from multiple dimensions, aiming to provide a uniform evaluation platform and inspire researchers to explore more effective fairness solutions via a set of implications.

Related papers

• A Benchmark for Fairness-Aware Graph Learning [58.515305543487386]
  We present an extensive benchmark on ten representative fairness-aware graph learning methods. Our in-depth analysis reveals key insights into the strengths and limitations of existing methods.
  arXiv  Detail & Related papers  (2024-07-16T18:43:43Z)
• Standardizing Your Training Process for Human Activity Recognition Models: A Comprehensive Review in the Tunable Factors [4.199844472131922]
  We provide an exhaustive review of contemporary deep learning research in the field of wearable human activity recognition (WHAR) Our findings suggest that a major trend is the lack of detail provided by model training protocols. With insights from the analyses, we define a novel integrated training procedure tailored to the WHAR model.
  arXiv  Detail & Related papers  (2024-01-10T17:45:28Z)
• Better Understanding Differences in Attribution Methods via Systematic Evaluations [57.35035463793008]
  Post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. We propose three novel evaluation schemes to more reliably measure the faithfulness of those methods. We use these evaluation schemes to study strengths and shortcomings of some widely used attribution methods over a wide range of models.
  arXiv  Detail & Related papers  (2023-03-21T14:24:58Z)
• Towards Better Understanding Attribution Methods [77.1487219861185]
  Post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. We propose three novel evaluation schemes to more reliably measure the faithfulness of those methods. We also propose a post-processing smoothing step that significantly improves the performance of some attribution methods.
  arXiv  Detail & Related papers  (2022-05-20T20:50:17Z)
• On Modality Bias Recognition and Reduction [70.69194431713825]
  We study the modality bias problem in the context of multi-modal classification. We propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned. Our method yields remarkable performance improvements compared with the baselines.
  arXiv  Detail & Related papers  (2022-02-25T13:47:09Z)
• Through the Data Management Lens: Experimental Analysis and Evaluation of Fair Classification [75.49600684537117]
  Data management research is showing an increasing presence and interest in topics related to data and algorithmic fairness. We contribute a broad analysis of 13 fair classification approaches and additional variants, over their correctness, fairness, efficiency, scalability, and stability. Our analysis highlights novel insights on the impact of different metrics and high-level approach characteristics on different aspects of performance.
  arXiv  Detail & Related papers  (2021-01-18T22:55:40Z)
• Recent Progress in Appearance-based Action Recognition [73.6405863243707]
  Action recognition is a task to identify various human actions in a video. Recent appearance-based methods have achieved promising progress towards accurate action recognition.
  arXiv  Detail & Related papers  (2020-11-25T10:18:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.