Don't Throw it Away! The Utility of Unlabeled Data in Fair Decision Making

• URL: http://arxiv.org/abs/2205.04790v2
• Date: Wed, 11 May 2022 14:06:55 GMT
• Title: Don't Throw it Away! The Utility of Unlabeled Data in Fair Decision Making
• Authors: Miriam Rateike, Ayan Majumdar, Olga Mineeva, Krishna P. Gummadi, Isabel Valera
• Abstract summary: We propose a novel method based on a variational autoencoder for practical fair decision-making. Our method learns an unbiased data representation leveraging both labeled and unlabeled data. Our method converges to the optimal (fair) policy according to the ground-truth with low variance.
• Score: 14.905698014932488
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decision making algorithms, in practice, are often trained on data that exhibits a variety of biases. Decision-makers often aim to take decisions based on some ground-truth target that is assumed or expected to be unbiased, i.e., equally distributed across socially salient groups. In many practical settings, the ground-truth cannot be directly observed, and instead, we have to rely on a biased proxy measure of the ground-truth, i.e., biased labels, in the data. In addition, data is often selectively labeled, i.e., even the biased labels are only observed for a small fraction of the data that received a positive decision. To overcome label and selection biases, recent work proposes to learn stochastic, exploring decision policies via i) online training of new policies at each time-step and ii) enforcing fairness as a constraint on performance. However, the existing approach uses only labeled data, disregarding a large amount of unlabeled data, and thereby suffers from high instability and variance in the learned decision policies at different times. In this paper, we propose a novel method based on a variational autoencoder for practical fair decision-making. Our method learns an unbiased data representation leveraging both labeled and unlabeled data and uses the representations to learn a policy in an online process. Using synthetic data, we empirically validate that our method converges to the optimal (fair) policy according to the ground-truth with low variance. In real-world experiments, we further show that our training approach not only offers a more stable learning process but also yields policies with higher fairness as well as utility than previous approaches.

Related papers

• Unlabeled Debiasing in Downstream Tasks via Class-wise Low Variance Regularization [13.773597081543185]
  We introduce a novel debiasing regularization technique based on the class-wise variance of embeddings. Our method does not require attribute labels and targets any attribute, thus addressing the shortcomings of existing debiasing methods.
  arXiv  Detail & Related papers  (2024-09-29T03:56:50Z)
• Partial-Label Regression [54.74984751371617]
  Partial-label learning is a weakly supervised learning setting that allows each training example to be annotated with a set of candidate labels. Previous studies on partial-label learning only focused on the classification setting where candidate labels are all discrete. In this paper, we provide the first attempt to investigate partial-label regression, where each training example is annotated with a set of real-valued candidate labels.
  arXiv  Detail & Related papers  (2023-06-15T09:02:24Z)
• Fairness and Bias in Truth Discovery Algorithms: An Experimental Analysis [7.575734557466221]
  Crowd workers may sometimes provide unreliable labels. Truth discovery (TD) algorithms are applied to determine the consensus labels from conflicting worker responses. We conduct a systematic study of the bias and fairness of TD algorithms.
  arXiv  Detail & Related papers  (2023-04-25T04:56:35Z)
• Dist-PU: Positive-Unlabeled Learning from a Label Distribution Perspective [89.5370481649529]
  We propose a label distribution perspective for PU learning in this paper. Motivated by this, we propose to pursue the label distribution consistency between predicted and ground-truth label distributions. Experiments on three benchmark datasets validate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-12-06T07:38:29Z)
• Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries [53.222218035435006]
  We use adversarial tools to optimize for queries that are discriminative and diverse. Our improvements achieve significantly more accurate membership inference than existing methods.
  arXiv  Detail & Related papers  (2022-10-19T17:46:50Z)
• Weighted Distillation with Unlabeled Examples [15.825078347452024]
  Distillation with unlabeled examples is a popular and powerful method for training deep neural networks in settings where the amount of labeled data is limited. This paper proposes a principled approach for addressing this issue based on a ''debiasing'' reweighting of the student's loss function tailored to the distillation training paradigm.
  arXiv  Detail & Related papers  (2022-10-13T04:08:56Z)
• Debiased Pseudo Labeling in Self-Training [77.83549261035277]
  Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets. To mitigate the requirement for labeled data, self-training is widely used in both academia and industry by pseudo labeling on readily-available unlabeled data. We propose Debiased, in which the generation and utilization of pseudo labels are decoupled by two independent heads.
  arXiv  Detail & Related papers  (2022-02-15T02:14:33Z)
• How to Leverage Unlabeled Data in Offline Reinforcement Learning [125.72601809192365]
  offline reinforcement learning (RL) can learn control policies from static datasets but, like standard RL methods, it requires reward annotations for every transition. One natural solution is to learn a reward function from the labeled data and use it to label the unlabeled data. We find that, perhaps surprisingly, a much simpler method that simply applies zero rewards to unlabeled data leads to effective data sharing.
  arXiv  Detail & Related papers  (2022-02-03T18:04:54Z)
• Learning with Proper Partial Labels [87.65718705642819]
  Partial-label learning is a kind of weakly-supervised learning with inexact labels. We show that this proper partial-label learning framework includes many previous partial-label learning settings. We then derive a unified unbiased estimator of the classification risk.
  arXiv  Detail & Related papers  (2021-12-23T01:37:03Z)
• Bias-Tolerant Fair Classification [20.973916494320246]
  label bias and selection bias are two reasons in data that will hinder the fairness of machine-learning outcomes. We propose a Bias-TolerantFAirRegularizedLoss (B-FARL) which tries to regain the benefits using data affected by label bias and selection bias. B-FARL takes the biased data as input, calls a model that approximates the one trained with fair but latent data, and thus prevents discrimination without constraints required.
  arXiv  Detail & Related papers  (2021-07-07T13:31:38Z)
• A Ranking Approach to Fair Classification [11.35838396538348]
  Algorithmic decision systems are increasingly used in areas such as hiring, school admission, or loan approval. In many scenarios, ground-truth labels are unavailable, and instead we have only access to imperfect labels as the result of human-made decisions. We propose a new fair ranking-based decision system, as an alternative to traditional classification algorithms.
  arXiv  Detail & Related papers  (2021-02-08T22:51:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.