Fair Pairs: Fairness-Aware Ranking Recovery from Pairwise Comparisons

• URL: http://arxiv.org/abs/2408.13034v1
• Date: Fri, 23 Aug 2024 12:46:16 GMT
• Title: Fair Pairs: Fairness-Aware Ranking Recovery from Pairwise Comparisons
• Authors: Georg Ahnert, Antonio Ferrara, Claudia Wagner,
• Abstract summary: We introduce the problem of fairness-aware ranking recovery from pairwise comparisons. We propose a group-conditioned accuracy measure which quantifies fairness of rankings recovered from pairwise comparisons.
• Score: 2.056289813004423
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pairwise comparisons based on human judgements are an effective method for determining rankings of items or individuals. However, as human biases perpetuate from pairwise comparisons to recovered rankings, they affect algorithmic decision making. In this paper, we introduce the problem of fairness-aware ranking recovery from pairwise comparisons. We propose a group-conditioned accuracy measure which quantifies fairness of rankings recovered from pairwise comparisons. We evaluate the impact of state-of-the-art ranking recovery algorithms and sampling approaches on accuracy and fairness of the recovered rankings, using synthetic and empirical data. Our results show that Fairness-Aware PageRank and GNNRank with FA*IR post-processing effectively mitigate existing biases in pairwise comparisons and improve the overall accuracy of recovered rankings. We highlight limitations and strengths of different approaches, and provide a Python package to facilitate replication and future work on fair ranking recovery from pairwise comparisons.

Related papers

• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Learning when to rank: Estimation of partial rankings from sparse, noisy comparisons [0.0]
  We develop a principled Bayesian methodology for learning partial rankings. Our framework is adaptable to any statistical ranking method. It gives a more parsimonious summary of the data than traditional ranking.
  arXiv  Detail & Related papers  (2025-01-05T11:04:30Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Explainable Disparity Compensation for Efficient Fair Ranking [0.3759936323189418]
  Ranking functions that are used in decision systems often produce disparate results for different populations because of bias in the underlying data. Recent compensatory measures have mostly focused on opaque transformations of the ranking functions to satisfy fairness guarantees. In this paper we propose easily explainable data-driven compensatory measures for ranking functions.
  arXiv  Detail & Related papers  (2023-07-25T09:12:50Z)
• Matched Pair Calibration for Ranking Fairness [2.580183306478581]
  We propose a test of fairness in score-based ranking systems called matched pair calibration. We show how our approach generalizes the fairness intuitions of calibration from a binary classification setting to ranking.
  arXiv  Detail & Related papers  (2023-06-06T15:32:30Z)
• Decreasing Annotation Burden of Pairwise Comparisons with Human-in-the-Loop Sorting: Application in Medical Image Artifact Rating [3.5314411880556063]
  Ranking by pairwise comparisons has shown improved reliability over ordinal classification. We propose a method for reducing the number of pairwise comparisons required to rank by a quantitative metric.
  arXiv  Detail & Related papers  (2022-02-10T04:02:45Z)
• Heuristic Search for Rank Aggregation with Application to Label Ranking [16.275063634853584]
  We propose an effective hybrid evolutionary ranking algorithm to solve the rank aggregation problem. The algorithm features a semantic crossover based on concordant pairs and a late acceptance local search reinforced by an efficient incremental evaluation technique. Experiments are conducted to assess the algorithm, indicating a highly competitive performance on benchmark instances.
  arXiv  Detail & Related papers  (2022-01-11T11:43:17Z)
• A Pre-processing Method for Fairness in Ranking [0.0]
  We propose a fair ranking framework that evaluates the order of training data in a pairwise manner. We show that our method outperforms the existing methods in the trade-off between accuracy and fairness over real-world datasets.
  arXiv  Detail & Related papers  (2021-10-29T02:55:32Z)
• Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisons [85.5955376526419]
  In rank aggregation problems, users exhibit various accuracy levels when comparing pairs of items. We propose an elimination-based active sampling strategy, which estimates the ranking of items via noisy pairwise comparisons. We prove that our algorithm can return the true ranking of items with high probability.
  arXiv  Detail & Related papers  (2021-10-08T13:51:55Z)
• Can Active Learning Preemptively Mitigate Fairness Issues? [66.84854430781097]
  dataset bias is one of the prevailing causes of unfairness in machine learning. We study whether models trained with uncertainty-based ALs are fairer in their decisions with respect to a protected class. We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD.
  arXiv  Detail & Related papers  (2021-04-14T14:20:22Z)
• Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking [74.46448041224247]
  We introduce the novel Logging-Policy Optimization Algorithm (LogOpt), which optimize the policy for logging data. LogOpt turns the counterfactual approach - which is indifferent to the logging policy - into an online approach, where the algorithm decides what rankings to display. We prove that, as an online evaluation method, LogOpt is unbiased w.r.t. position and item-selection bias, unlike existing interleaving methods.
  arXiv  Detail & Related papers  (2020-07-24T18:05:58Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)
• SetRank: A Setwise Bayesian Approach for Collaborative Ranking from Implicit Feedback [50.13745601531148]
  We propose a novel setwise Bayesian approach for collaborative ranking, namely SetRank, to accommodate the characteristics of implicit feedback in recommender system. Specifically, SetRank aims at maximizing the posterior probability of novel setwise preference comparisons. We also present the theoretical analysis of SetRank to show that the bound of excess risk can be proportional to $sqrtM/N$.
  arXiv  Detail & Related papers  (2020-02-23T06:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.