Stability and Multigroup Fairness in Ranking with Uncertain Predictions

• URL: http://arxiv.org/abs/2402.09326v1
• Date: Wed, 14 Feb 2024 17:17:05 GMT
• Title: Stability and Multigroup Fairness in Ranking with Uncertain Predictions
• Authors: Siddartha Devic, Aleksandra Korolova, David Kempe, Vatsal Sharan
• Abstract summary: Our work considers ranking functions: maps from individual predictions for a classification task to distributions over rankings. We focus on two aspects of ranking functions: stability to perturbations in predictions and fairness towards both individuals and subgroups. Our work demonstrates that uncertainty aware rankings naturally interpolate between group and individual level fairness guarantees.
• Score: 61.76378420347408
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Rankings are ubiquitous across many applications, from search engines to hiring committees. In practice, many rankings are derived from the output of predictors. However, when predictors trained for classification tasks have intrinsic uncertainty, it is not obvious how this uncertainty should be represented in the derived rankings. Our work considers ranking functions: maps from individual predictions for a classification task to distributions over rankings. We focus on two aspects of ranking functions: stability to perturbations in predictions and fairness towards both individuals and subgroups. Not only is stability an important requirement for its own sake, but -- as we show -- it composes harmoniously with individual fairness in the sense of Dwork et al. (2012). While deterministic ranking functions cannot be stable aside from trivial scenarios, we show that the recently proposed uncertainty aware (UA) ranking functions of Singh et al. (2021) are stable. Our main result is that UA rankings also achieve multigroup fairness through successful composition with multiaccurate or multicalibrated predictors. Our work demonstrates that UA rankings naturally interpolate between group and individual level fairness guarantees, while simultaneously satisfying stability guarantees important whenever machine-learned predictions are used.

Related papers

• Interpreting Predictive Probabilities: Model Confidence or Human Label Variation? [27.226997687210044]
  We identify two main perspectives that drive starkly different evaluation protocols. We discuss their merits and limitations, and take the position that both are crucial for trustworthy and fair NLP systems. We recommend tools and highlight exciting directions towards models with disentangled representations of uncertainty about predictions and uncertainty about human labels.
  arXiv  Detail & Related papers  (2024-02-25T15:00:13Z)
• Ranking a Set of Objects using Heterogeneous Workers: QUITE an Easy Problem [54.90613714264689]
  We focus on the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of unequal workers. We propose QUITE, a non-adaptive ranking algorithm that jointly estimates workers' reliabilities and qualities of objects.
  arXiv  Detail & Related papers  (2023-10-03T12:42:13Z)
• Fairness in Ranking under Disparate Uncertainty [24.401219403555814]
  We argue that ranking can introduce unfairness if the uncertainty of the underlying relevance model differs between groups of options. We propose Equal-Opportunity Ranking (EOR) as a new fairness criterion for ranking. We show that EOR corresponds to a group-wise fair lottery among the relevant options even in the presence of disparate uncertainty.
  arXiv  Detail & Related papers  (2023-09-04T13:49:48Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Learning to Rank when Grades Matter [11.981942948477236]
  Graded labels are ubiquitous in real-world learning-to-rank applications. Traditional learning-to-rank techniques ignore predicting actual grades. We propose a multiobjective formulation to jointly optimize both ranking and grade predictions.
  arXiv  Detail & Related papers  (2023-06-14T17:30:02Z)
• Fairness in Matching under Uncertainty [78.39459690570531]
  algorithmic two-sided marketplaces have drawn attention to the issue of fairness in such settings. We axiomatize a notion of individual fairness in the two-sided marketplace setting which respects the uncertainty in the merits. We design a linear programming framework to find fair utility-maximizing distributions over allocations.
  arXiv  Detail & Related papers  (2023-02-08T00:30:32Z)
• Estimating Confidence of Predictions of Individual Classifiers and Their Ensembles for the Genre Classification Task [0.0]
  Genre identification is a subclass of non-topical text classification. Nerve models based on pre-trained transformers, such as BERT or XLM-RoBERTa, demonstrate SOTA results in many NLP tasks.
  arXiv  Detail & Related papers  (2022-06-15T09:59:05Z)
• Learning Equilibria in Matching Markets from Bandit Feedback [139.29934476625488]
  We develop a framework and algorithms for learning stable market outcomes under uncertainty. Our work takes a first step toward elucidating when and how stable matchings arise in large, data-driven marketplaces.
  arXiv  Detail & Related papers  (2021-08-19T17:59:28Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.