The Role of Relevance in Fair Ranking

• URL: http://arxiv.org/abs/2305.05608v2
• Date: Tue, 6 Jun 2023 15:02:00 GMT
• Title: The Role of Relevance in Fair Ranking
• Authors: Aparna Balagopalan, Abigail Z. Jacobs, Asia Biega
• Abstract summary: We argue that relevance scores should satisfy a set of desired criteria in order to guide fairness interventions. We then empirically show that not all of these criteria are met in a case study of relevance inferred from biased user click data. Our analyses and results surface the pressing need for new approaches to relevance collection and generation.
• Score: 1.5469452301122177
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online platforms mediate access to opportunity: relevance-based rankings create and constrain options by allocating exposure to job openings and job candidates in hiring platforms, or sellers in a marketplace. In order to do so responsibly, these socially consequential systems employ various fairness measures and interventions, many of which seek to allocate exposure based on worthiness. Because these constructs are typically not directly observable, platforms must instead resort to using proxy scores such as relevance and infer them from behavioral signals such as searcher clicks. Yet, it remains an open question whether relevance fulfills its role as such a worthiness score in high-stakes fair rankings. In this paper, we combine perspectives and tools from the social sciences, information retrieval, and fairness in machine learning to derive a set of desired criteria that relevance scores should satisfy in order to meaningfully guide fairness interventions. We then empirically show that not all of these criteria are met in a case study of relevance inferred from biased user click data. We assess the impact of these violations on the estimated system fairness and analyze whether existing fairness interventions may mitigate the identified issues. Our analyses and results surface the pressing need for new approaches to relevance collection and generation that are suitable for use in fair ranking.

Related papers

• Ranking Policy Learning via Marketplace Expected Value Estimation From Observational Data [8.985446716914515]
  We study the problem of learning a ranking policy for search or recommendation engines in a two-sided e-commerce marketplace. As a value allocation mechanism, the ranking policy allocates retrieved items to the designated slots. We build empirical estimates for the expected reward of the marketplace from observational data.
  arXiv  Detail & Related papers  (2024-10-06T17:53:44Z)
• Fairness meets Cross-Domain Learning: a new perspective on Models and Metrics [80.07271410743806]
  We study the relationship between cross-domain learning (CD) and model fairness. We introduce a benchmark on face and medical images spanning several demographic groups as well as classification and localization tasks. Our study covers 14 CD approaches alongside three state-of-the-art fairness algorithms and shows how the former can outperform the latter.
  arXiv  Detail & Related papers  (2023-03-25T09:34:05Z)
• DualFair: Fair Representation Learning at Both Group and Individual Levels via Contrastive Self-supervision [73.80009454050858]
  This work presents a self-supervised model, called DualFair, that can debias sensitive attributes like gender and race from learned representations. Our model jointly optimize for two fairness criteria - group fairness and counterfactual fairness.
  arXiv  Detail & Related papers  (2023-03-15T07:13:54Z)
• Fairness in Matching under Uncertainty [78.39459690570531]
  algorithmic two-sided marketplaces have drawn attention to the issue of fairness in such settings. We axiomatize a notion of individual fairness in the two-sided marketplace setting which respects the uncertainty in the merits. We design a linear programming framework to find fair utility-maximizing distributions over allocations.
  arXiv  Detail & Related papers  (2023-02-08T00:30:32Z)
• Fairness in Contextual Resource Allocation Systems: Metrics and Incompatibility Results [7.705334602362225]
  We study systems that allocate scarce resources to satisfy basic needs, such as homeless services that provide housing. These systems often support communities disproportionately affected by systemic racial, gender, or other injustices. We propose a framework for evaluating fairness in contextual resource allocation systems inspired by fairness metrics in machine learning.
  arXiv  Detail & Related papers  (2022-12-04T02:30:58Z)
• Fair Ranking with Noisy Protected Attributes [25.081136190260015]
  We study the fair-ranking problem under a model where socially-salient attributes of items are randomly and independently perturbed. We present a fair-ranking framework that incorporates group fairness requirements along with probabilistic information about perturbations in socially-salient attributes.
  arXiv  Detail & Related papers  (2022-11-30T15:22:28Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Fair ranking: a critical review, challenges, and future directions [4.126546022406797]
  Large "fair ranking" research literature has been developed around making these systems fair to the individuals, providers, or content that are being ranked. This work provides a critical overview of this literature, detailing the often context-specific concerns that such an approach misses. We then provide a path forward for a more holistic and impact-oriented fair ranking research agenda.
  arXiv  Detail & Related papers  (2022-01-29T21:26:04Z)
• Towards a Fairness-Aware Scoring System for Algorithmic Decision-Making [35.21763166288736]
  We propose a general framework to create data-driven fairness-aware scoring systems. We show that the proposed framework provides practitioners or policymakers great flexibility to select their desired fairness requirements.
  arXiv  Detail & Related papers  (2021-09-21T09:46:35Z)
• Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach [131.20444904674494]
  We tackle the problem of measuring group fairness under unawareness of sensitive attributes. We show that quantification approaches are particularly suited to tackle the fairness-under-unawareness problem.
  arXiv  Detail & Related papers  (2021-09-17T13:45:46Z)
• Through the Data Management Lens: Experimental Analysis and Evaluation of Fair Classification [75.49600684537117]
  Data management research is showing an increasing presence and interest in topics related to data and algorithmic fairness. We contribute a broad analysis of 13 fair classification approaches and additional variants, over their correctness, fairness, efficiency, scalability, and stability. Our analysis highlights novel insights on the impact of different metrics and high-level approach characteristics on different aspects of performance.
  arXiv  Detail & Related papers  (2021-01-18T22:55:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.