Scoring Is Not Enough: Addressing Gaps in Utility-fairness Trade-offs for Ranking

• URL: http://arxiv.org/abs/2606.26369v1
• Date: Wed, 24 Jun 2026 20:36:42 GMT
• Title: Scoring Is Not Enough: Addressing Gaps in Utility-fairness Trade-offs for Ranking
• Abstract summary: We show that scoring is sub-optimal in achieving all utility-fairness trade-offs.<n>We empirically demonstrate that semi-greedy post-processing has the potential to achieve much better trade-offs.
• Score: 7.560688419767115
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scoring functions are used to represent the relevance of individual documents. In modern information retrieval or recommendation systems, they are often learned from data and play a pivotal role in ranking sets of documents or items in a way that maximizes utility to a query or user. With the recent interest in algorithmic fairness, the success of scoring has naturally led to methods that learn scores that simultaneously trade off fairness and utility. In this work, we show that in stark contrast with utility-centric objectives, scoring is sub-optimal in achieving all utility-fairness trade-offs. We establish this with a series of counter-examples with a generic fairness formulation. We show that the issue persists whether we have a deterministic scoring function or a randomized one, or whether we measure fairness at the scope of a single query or across multiple queries. On the positive side, we empirically demonstrate that semi-greedy post-processing has the potential to achieve much better trade-offs, often approaching the ideal of exhaustive post-processing in a tractable way.

Related papers

• What's in a Query: Polarity-Aware Distribution-Based Fair Ranking [15.602253525743798]
  We propose new measures for attention distribution-based fairness in ranking.<n>We prove that group fairness is upper-bounded by individual fairness.<n>We find that prior research in amortized fair ranking ignores critical information about queries.
  arXiv  Detail & Related papers  (2025-02-17T04:38:36Z)
• A Benchmark for Fairness-Aware Graph Learning [58.515305543487386]
  We present an extensive benchmark on ten representative fairness-aware graph learning methods. Our in-depth analysis reveals key insights into the strengths and limitations of existing methods.
  arXiv  Detail & Related papers  (2024-07-16T18:43:43Z)
• Utility-Fairness Trade-Offs and How to Find Them [14.1278892335105]
  We introduce two utility-fairness trade-offs: the Data-Space and Label-Space Trade-off. We propose U-FaTE, a method to numerically quantify the trade-offs for a given prediction task and group fairness definition from data samples. An extensive evaluation of fair representation learning methods and representations from over 1000 pre-trained models revealed that most current approaches are far from the estimated and achievable fairness-utility trade-offs.
  arXiv  Detail & Related papers  (2024-04-15T04:43:53Z)
• Evaluating the Fairness of Discriminative Foundation Models in Computer Vision [51.176061115977774]
  We propose a novel taxonomy for bias evaluation of discriminative foundation models, such as Contrastive Language-Pretraining (CLIP) We then systematically evaluate existing methods for mitigating bias in these models with respect to our taxonomy. Specifically, we evaluate OpenAI's CLIP and OpenCLIP models for key applications, such as zero-shot classification, image retrieval and image captioning.
  arXiv  Detail & Related papers  (2023-10-18T10:32:39Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Sampling Individually-Fair Rankings that are Always Group Fair [9.333939443470944]
  A fair ranking task asks to rank a set of items to maximize utility subject to satisfying group-fairness constraints. Recent works identify uncertainty in the utilities of items as a primary cause of unfairness. We give an efficient algorithm that samples rankings from an individually-fair distribution while ensuring that every output ranking is group fair.
  arXiv  Detail & Related papers  (2023-06-21T01:26:34Z)
• The Role of Relevance in Fair Ranking [1.5469452301122177]
  We argue that relevance scores should satisfy a set of desired criteria in order to guide fairness interventions. We then empirically show that not all of these criteria are met in a case study of relevance inferred from biased user click data. Our analyses and results surface the pressing need for new approaches to relevance collection and generation.
  arXiv  Detail & Related papers  (2023-05-09T16:58:23Z)
• Estimation of Fair Ranking Metrics with Incomplete Judgments [70.37717864975387]
  We propose a sampling strategy and estimation technique for four fair ranking metrics. We formulate a robust and unbiased estimator which can operate even with very limited number of labeled items.
  arXiv  Detail & Related papers  (2021-08-11T10:57:00Z)
• How Many Data Points is a Prompt Worth? [106.76346863035786]
  Proponents of prompting argue that it provides a method for injecting task-specific guidance. We compare prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. Results show that prompting is often worth 100s of data points on average across classification tasks.
  arXiv  Detail & Related papers  (2021-03-15T16:10:23Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)
• Controlling Fairness and Bias in Dynamic Learning-to-Rank [31.41843594914603]
  We propose a learning algorithm that ensures notions of amortized group fairness, while simultaneously learning the ranking function from implicit feedback data. The algorithm takes the form of a controller that integrates unbiased estimators for both fairness and utility. In addition to its rigorous theoretical foundation and convergence guarantees, we find empirically that the algorithm is highly practical and robust.
  arXiv  Detail & Related papers  (2020-05-29T17:57:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.