On the Impact of Data Quality on Image Classification Fairness

• URL: http://arxiv.org/abs/2305.01595v1
• Date: Tue, 2 May 2023 16:54:23 GMT
• Title: On the Impact of Data Quality on Image Classification Fairness
• Authors: Aki Barry and Lei Han and Gianluca Demartini
• Abstract summary: We measure key fairness metrics across a range of algorithms over multiple image classification datasets. We describe noise in the labels as inaccuracies in the labelling of the data in the training set and noise in the data as distortions in the data. By adding noise to the original datasets, we can explore the relationship between the quality of the training data and the fairness of the output of the models trained on that data.
• Score: 11.329873246415797
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the proliferation of algorithmic decision-making, increased scrutiny has been placed on these systems. This paper explores the relationship between the quality of the training data and the overall fairness of the models trained with such data in the context of supervised classification. We measure key fairness metrics across a range of algorithms over multiple image classification datasets that have a varying level of noise in both the labels and the training data itself. We describe noise in the labels as inaccuracies in the labelling of the data in the training set and noise in the data as distortions in the data, also in the training set. By adding noise to the original datasets, we can explore the relationship between the quality of the training data and the fairness of the output of the models trained on that data.

Related papers

• Fair-OBNC: Correcting Label Noise for Fairer Datasets [9.427445881721814]
  biases in the training data are sometimes related to label noise. Models trained on such biased data may perpetuate or even aggravate the biases with respect to sensitive information. We propose Fair-OBNC, a label noise correction method with fairness considerations.
  arXiv  Detail & Related papers  (2024-10-08T17:18:18Z)
• In-Context Probing Approximates Influence Function for Data Valuation [16.404477234171733]
  We show that data valuation through in-context probing approximates influence functions for selecting training data. Our empirical findings show that in-context probing and gradient-based influence frameworks are similar in how they rank training data.
  arXiv  Detail & Related papers  (2024-07-17T02:06:56Z)
• Soft Curriculum for Learning Conditional GANs with Noisy-Labeled and Uncurated Unlabeled Data [70.25049762295193]
  We introduce a novel conditional image generation framework that accepts noisy-labeled and uncurated data during training. We propose soft curriculum learning, which assigns instance-wise weights for adversarial training while assigning new labels for unlabeled data. Our experiments show that our approach outperforms existing semi-supervised and label-noise robust methods in terms of both quantitative and qualitative performance.
  arXiv  Detail & Related papers  (2023-07-17T08:31:59Z)
• Systematic analysis of the impact of label noise correction on ML Fairness [0.0]
  We develop an empirical methodology to evaluate the effectiveness of label noise correction techniques in ensuring the fairness of models trained on biased datasets. Our results suggest that the Hybrid Label Noise Correction method achieves the best trade-off between predictive performance and fairness.
  arXiv  Detail & Related papers  (2023-06-28T08:08:14Z)
• On-the-fly Denoising for Data Augmentation in Natural Language Understanding [101.46848743193358]
  We propose an on-the-fly denoising technique for data augmentation that learns from soft augmented labels provided by an organic teacher model trained on the cleaner original data. Our method can be applied to general augmentation techniques and consistently improve the performance on both text classification and question-answering tasks.
  arXiv  Detail & Related papers  (2022-12-20T18:58:33Z)
• Representation Learning for the Automatic Indexing of Sound Effects Libraries [79.68916470119743]
  We show that a task-specific but dataset-independent representation can successfully address data issues such as class imbalance, inconsistent class labels, and insufficient dataset size. Detailed experimental results show the impact of metric learning approaches and different cross-dataset training methods on representational effectiveness.
  arXiv  Detail & Related papers  (2022-08-18T23:46:13Z)
• Assessing the Quality of the Datasets by Identifying Mislabeled Samples [14.881597737762316]
  We propose a novel statistic -- noise score -- as a measure for the quality of each data point to identify mislabeled samples. In our work, we use the representations derived by the inference network of data quality supervised variational autoencoder (AQUAVS) We validate our proposed statistic through experimentation by corrupting MNIST, FashionMNIST, and CIFAR10/100 datasets.
  arXiv  Detail & Related papers  (2021-09-10T17:14:09Z)
• Improving filling level classification with adversarial training [90.01594595780928]
  We investigate the problem of classifying - from a single image - the level of content in a cup or a drinking glass. We use adversarial training in a generic source dataset and then refine the training with a task-specific dataset. We show that transfer learning with adversarial training in the source domain consistently improves the classification accuracy on the test set.
  arXiv  Detail & Related papers  (2021-02-08T08:32:56Z)
• Unsupervised Domain Adaptation for Acoustic Scene Classification Using Band-Wise Statistics Matching [69.24460241328521]
  Machine learning algorithms can be negatively affected by mismatches between training (source) and test (target) data distributions. We propose an unsupervised domain adaptation method that consists of aligning the first- and second-order sample statistics of each frequency band of target-domain acoustic scenes to the ones of the source-domain training dataset. We show that the proposed method outperforms the state-of-the-art unsupervised methods found in the literature in terms of both source- and target-domain classification accuracy.
  arXiv  Detail & Related papers  (2020-04-30T23:56:05Z)
• On the Role of Dataset Quality and Heterogeneity in Model Confidence [27.657631193015252]
  Safety-critical applications require machine learning models that output accurate and calibrated probabilities. Uncalibrated deep networks are known to make over-confident predictions. We study the impact of dataset quality by studying the impact of dataset size and the label noise on the model confidence.
  arXiv  Detail & Related papers  (2020-02-23T05:13:12Z)
• Learning with Out-of-Distribution Data for Audio Classification [60.48251022280506]
  We show that detecting and relabelling certain OOD instances, rather than discarding them, can have a positive effect on learning. The proposed method is shown to improve the performance of convolutional neural networks by a significant margin.
  arXiv  Detail & Related papers  (2020-02-11T21:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.