Fairly Accurate: Learning Optimal Accuracy vs. Fairness Tradeoffs for Hate Speech Detection

• URL: http://arxiv.org/abs/2204.07661v1
• Date: Fri, 15 Apr 2022 22:11:25 GMT
• Title: Fairly Accurate: Learning Optimal Accuracy vs. Fairness Tradeoffs for Hate Speech Detection
• Authors: Venelin Kovatchev, Soumyajit Gupta, Matthew Lease
• Abstract summary: We introduce a differentiable measure that enables direct optimization of group fairness in model training. We evaluate our methods on the specific task of hate speech detection. Empirical results across convolutional, sequential, and transformer-based neural architectures show superior empirical accuracy vs. fairness trade-offs over prior work.
• Score: 8.841221697099687
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work has emphasized the importance of balancing competing objectives in model training (e.g., accuracy vs. fairness, or competing measures of fairness). Such trade-offs reflect a broader class of multi-objective optimization (MOO) problems in which optimization methods seek Pareto optimal trade-offs between competing goals. In this work, we first introduce a differentiable measure that enables direct optimization of group fairness (specifically, balancing accuracy across groups) in model training. Next, we demonstrate two model-agnostic MOO frameworks for learning Pareto optimal parameterizations over different groups of neural classification models. We evaluate our methods on the specific task of hate speech detection, in which prior work has shown lack of group fairness across speakers of different English dialects. Empirical results across convolutional, sequential, and transformer-based neural architectures show superior empirical accuracy vs. fairness trade-offs over prior work. More significantly, our measure enables the Pareto machinery to ensure that each architecture achieves the best possible trade-off between fairness and accuracy w.r.t. the dataset, given user-prescribed error tolerance bounds.

Related papers

• Fair Bilevel Neural Network (FairBiNN): On Balancing fairness and accuracy via Stackelberg Equilibrium [0.3350491650545292]
  Current methods for mitigating bias often result in information loss and an inadequate balance between accuracy and fairness. We propose a novel methodology grounded in bilevel optimization principles. Our deep learning-based approach concurrently optimize for both accuracy and fairness objectives.
  arXiv  Detail & Related papers  (2024-10-21T18:53:39Z)
• Fairness-Aware Meta-Learning via Nash Bargaining [63.44846095241147]
  We introduce a two-stage meta-learning framework to address issues of group-level fairness in machine learning. The first stage involves the use of a Nash Bargaining Solution (NBS) to resolve hypergradient conflicts and steer the model. We show empirical effects across various fairness objectives in six key fairness datasets and two image classification tasks.
  arXiv  Detail & Related papers  (2024-06-11T07:34:15Z)
• Boosting Fair Classifier Generalization through Adaptive Priority Reweighing [59.801444556074394]
  A performance-promising fair algorithm with better generalizability is needed. This paper proposes a novel adaptive reweighing method to eliminate the impact of the distribution shifts between training and test data on model generalizability.
  arXiv  Detail & Related papers  (2023-09-15T13:04:55Z)
• Fair Few-shot Learning with Auxiliary Sets [53.30014767684218]
  In many machine learning (ML) tasks, only very few labeled data samples can be collected, which can lead to inferior fairness performance. In this paper, we define the fairness-aware learning task with limited training samples as the emphfair few-shot learning problem. We devise a novel framework that accumulates fairness-aware knowledge across different meta-training tasks and then generalizes the learned knowledge to meta-test tasks.
  arXiv  Detail & Related papers  (2023-08-28T06:31:37Z)
• Optimizing fairness tradeoffs in machine learning with multiobjective meta-models [0.913755431537592]
  We present a flexible framework for defining the fair machine learning task as a weighted classification problem with multiple cost functions. We use multiobjective optimization to define the sample weights used in model training for a given machine learner, and adapt the weights to optimize multiple metrics of fairness and accuracy. On a set of real-world problems, this approach outperforms current state-of-the-art methods by finding solution sets with preferable error/fairness trade-offs.
  arXiv  Detail & Related papers  (2023-04-21T13:42:49Z)
• DualFair: Fair Representation Learning at Both Group and Individual Levels via Contrastive Self-supervision [73.80009454050858]
  This work presents a self-supervised model, called DualFair, that can debias sensitive attributes like gender and race from learned representations. Our model jointly optimize for two fairness criteria - group fairness and counterfactual fairness.
  arXiv  Detail & Related papers  (2023-03-15T07:13:54Z)
• Mitigating Unfairness via Evolutionary Multi-objective Ensemble Learning [0.8563354084119061]
  Optimising one or several fairness measures may sacrifice or deteriorate other measures. A multi-objective evolutionary learning framework is used to simultaneously optimise several metrics. Our proposed algorithm can provide decision-makers with better tradeoffs among accuracy and multiple fairness metrics.
  arXiv  Detail & Related papers  (2022-10-30T06:34:10Z)
• Learning Optimal Fair Classification Trees: Trade-offs Between Interpretability, Fairness, and Accuracy [7.215903549622416]
  We propose a mixed integer optimization framework for learning optimal classification trees. We benchmark our method against state-of-the-art approaches for fair classification on popular datasets. Our method consistently finds decisions with almost full parity, while other methods rarely do.
  arXiv  Detail & Related papers  (2022-01-24T19:47:10Z)
• Deep F-measure Maximization for End-to-End Speech Understanding [52.36496114728355]
  We propose a differentiable approximation to the F-measure and train the network with this objective using standard backpropagation. We perform experiments on two standard fairness datasets, Adult, Communities and Crime, and also on speech-to-intent detection on the ATIS dataset and speech-to-image concept classification on the Speech-COCO dataset. In all four of these tasks, F-measure results in improved micro-F1 scores, with absolute improvements of up to 8% absolute, as compared to models trained with the cross-entropy loss function.
  arXiv  Detail & Related papers  (2020-08-08T03:02:27Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.